Overview of Different Web Structures

Overview of Different Web StructuresDark Inter pelf, Dark net ( commit Sharing), Turbo10, Meta-Search railway locomotiveMansi Iyengar (SCU ID W1170603)This project is tar pressed for the re anticipateers to gain insight into the diametrical sack up social structures. The primary focus being involved net and profane net.It as well as throws light on bear d receive sharing in phantom net along with meta take c ar engines apply by them.This report is based on dark net, metasearch engines and shoot down sharing mechanism. It has been categorize in the form of chapters.Each chapter gives us the below informationChapter 1 tells us about the different structures in the entanglementChapter 2 provides an all overview of dark net and TORChapter 3 describes the charge up sharing mechanism in dark net and putting surfacely used approachesChapter 4 focuses on operation of meta search engineChapter 5 talks about Turbo10 search engine for deep netThe linkage of each web paginate is referred by websites structure. take aim for a website having high rate of web pages. In such scenario, crawlers should have the ease to find the subpages. fancy 1 Web structure 1Figure 1 portrays entire web structure that comprises ofSurface webDeep webDark webSurface webThe traditional internet we use for tryless life is the world all-inclusive web.Represented by www.Deep webDeep web provides the hidden part of the internet. Search engine do not index these. Thus deep web stands apart from the traditional web. more or less comp singlents of deep web include email, online banking, on demand videosDark webDark web is different from above traditional web types.Dark net stands for a network accessed via specific softw be or configuration using protocol that are not standardized.For everyday life, we use regular search engines such as google. The exists websites that cannot be looked up on google. For such websites need a feature application in ensnare to access. Such webs ites are known as dark web.Dark net comprises of decentralized computers that collaborate in tandem to send information from origin to the tar overhear. Some of the prevalent dark net are Limewire, Gnucleus. Users are divisions that share information. Hosts are the computers that help share the information.Dark web is overly known as dark net or dark internet.For ease of understanding we can consider software named as Tor. Tor provides the functionality wherein the user cannot be identified on traditional websites.TORTor also provides sites that have hidden services. Basically the hidden or anonymous feature is leveraged to masquerade where server is situated. Tor makes it almost impossible to trace the members accessing website.Tor is not fool proof and comprises of some flaws. These pertain to security configuration. This gap can be used by officials to determine the real server location. The popularity of TOR is extremely high and hence most commonly used. whence the focus on this example.Features of dark netHowever, dark nets provide additional facilities different from above. These may include collaborative effort of handling non-licensed software and contentContent found on dark netOn two fronts dark net facilitates anonymity member visiting website and also for website itself. Governments too use dark net. For example, in order to gain entropy bypassing and censorship, Tor provides that optionDark net too can be used by politicians. This is serves as a platform to mediate and take end. Secrets are revealed by whistle blowers on certain sites so that they can avoid being traced. Webpages such as Strongbox help serve such purpose.This leads to establishment of websites that traditionally are not used. They can be used for anti-social purpose. For example, stolen credit card information, unlawful currency, drugs and weapons are found here. Gambling activities too are provided.Illegal drugs and goods, part of commercial dark market are made popu lar by silk route and diabolic market. These are attacked by legal teams.Alternate hacking services are sold. These are at group level or individualistic level. This has led to cyber-crimes and parry cyber investigation. Government has started looking into these using tools procured from Procedia Computer Science journal. Denial of service attacks too have been made through Dark Web. 4File SharingFile sharing involves the method of dividing or enabling digital media access such as audio files, video files, programs or eBooks.There are multiple ship canal to attain file sharing. Generally used approaches of storage, transmission include web url hyperlinked docs., p2p networks and centralized servers. 3Figure 2 file sharingThe above calculate shows us how finger shows how file sharing can be achieved between access controlled digital systems like private, public and invited.P2P file sharing surgical procedureComputers or nodes are connected with each other as part of P2P network. The members have the ability to share or download information immediately via the web.Figure 3 Peer to Peer networkAbove trope shows con feederate to peer network. Based on the model of self-server and client model, P2P is set up in general.for example, when the member connects to the P2P client and after initiating file download, he gets connected with other members downloading the information. In other words, others peers serve as file server. The initial member also acts as file server when other peer starts to download the file. Eventually this leads to augmenting of download speed. Some commonly used lushs for P2P are bit flood tide and Torrent. Other popular P2P networks being BearShare and LimeWire. 2OnionShareConsider App called OnionShare that is avail fitting for multiple Operating Systems. This is a P2P program. Entire information from TOR anonymity network is sent through this file sharing program.consider regular file sharing system needs faith. Law regulators can tap into these.Tor helps bypass third party, thither is direct sharing of file from one person to another via anonymous networkFigure 4 OnionShareThe above figure describes OnionShare process. After Tor instance is running, OnionShare will run. The beginning part is by starting a topical anaesthetic web server that is viewable as tor hidden service, generating web link that cannot be guessed out. This web link is used to download the file.The web link is then provided to the recipient person via secure methodthe webserver is cancelled The moment the receipt downloads the file,Now the file becomes inaccessible to anyone. These is scope to expand this further by having a continuous running server in order to share with multiple recipients 5BitTorrent File Sharing ProcessBackgroundBitTorrent file sharing protocol for dark net. BitTorrent is a peer-to-peer file sharing protocol and the most popular one according to a reportby ( two 1008). In the same report one can read that BitTorre nt takesup a substantial amount of all the internet traffic in the world, between 27 and 55 percent depending on geographical location. This makes BitTorrent not only the most used protocol for file sharing, but of all application layer protocols.As per Schulse and Mochalski , Bit Torrent is popular file sharing protocol. This is also used by Darknet.BitTorrent breaks data into parallel littler chunks instead of single chunk, enhancing reliability and reducing vulnerabilityApproaches for bit torrent file sharingProvided below are two approaches for bit torrent file sharingTorrent file approachTFigure 5 Torrent file sharing approachA torrent file is setup that has tracker, file contents. This file is smaller than original file. Seed is the uploader and leech is the downloader.Above figure describes the torrent file sharing approach. User first downloads the torrent. From torrent the set of peers is found. From the network, file pieces are exchanged. Files are treated instead of atomi c blocks but as smaller chunks.Advantage is that the bandwidth is spread among peer as against just the seeds.The tracker for a torrent provides peers a peer list. In this manner, bit torrent communication takes place, which is the same process involved for seeders and leachesDTH approachThere is a second approach that bypasses the tracker. This provides greater anonymity. This approach is performed without generating torrent file.For this purpose, bit torrent protocol ends up implementing DHT i.e. distributed hash table.Figure 6 DTH file sharing approachThe above figure describes DTH file sharing approach. DHT provides a set of key, value saved in it. The working is uniform to how has table works. Nodes have ability to detect noes based on the key. There is unique identification for each node. For this the communication utilized is User entropygram Protocol (UDP).The make is that there no centralized trackers. However, the track list is provided by peers that send peer list. tr aditionalistic torrent file has web link to a tracker. This tonicity is by passed in second approach, wherein DHT web link is used. DHT network is entered into by the peer using the URL. From the entry node, it can search crossways other networks for files and peer connection for file sharing. 6The results of other databases are combined in meta search engine. It takes up the concatenation of motley(a) results from various search engines and associate then to various resources. 1Virtual databases are generated from the mirrors of the physical database results of other search engines. These virtual databases are generated from the meta search engines. The searches are concentrated over certain criteria. 2BackgroundSearching the web with multiple search engines was the issue tackled by researchers. One of the outputs was meta search engine. Search Savvy was the first engine discovered and used by Daniel Dreilinger from Colorado deposit University. This engine looks over the resu lts from 20 different directories and search engines at once. Meta search engine crawler discovered by Eric Selberg at Washington University. It used its own search engine syntax and searched over 6 search engines.Dogpile formed at University of Pittsburgh and Pennsylvania State University measured ranking and overlap of various search engines. This proved to have one of the best results. 3AdvantagesMore information and search coverage can be achieved by using meta search engines. This helps the user to get faster and accurate results. The query of the user results is generated in unique ways by using the indexes which are aggregated by search engines. The input effort for a meta search engine and normal search engine are the same but the results retrieved are more. They also reduce the effort of users to type on various search engines to find file and resources.DisadvantagesTranslation of the query format or understanding the query forms can be an issue in the meta search engine. A ll the results sets of a query are not given to a user as the links generated from a meta search engine are limited. The links generated are limited. Promoted websites are given higher priority over other websites.It is probable that the user will get multiple results from the search engine. It may not be stream lined. This is e redundantly an issue due to more coverage over the queried topic. The users find it difficult to use meta search. They might not be able to get any precise information.OperationThe above figure describes the meta search operation. The search engine receives a query from the user.The key parts are as belowBroker The query needs a pre-processing as each search engine has a specific format of the data being fed in. As they depend on different operators and they do not share the same syntax. The output is a series of ordered set documents.Rank the documents are raked post identifying the result pages and also in the order of the most relevant links. 4The input query is passed to the search engines database. It creates a virtual database. This helps to combine data from various sources. Duplicates can be generated as various search-engines have different methods of indexing the data. This output data is then processed by the meta-search engine. The revised list is produced for the user. They would act in the following ways.1) Access to the meta-search engine reference is provided which also includes the private access to the database system The changes made on the database system would be captured.2) In a non-cooperative way, the access to the interface may or may not be provided.3) Meta search engine is denied access to the search engine.Architecture of rankingAbove figure describes the architecture of the meta search engine. It consists of the users query being sent to the meta search engine. It contacts the other search engines its connected to. They process the results and generate resources. These results are then preprocessed and th en given back to the meta search engine as a response. This collective response is given to the user.FusionFusion is a process for data filtering. It helps to build up efficient results.Collection FusionUnrelated data is indexed via search engines it is dealt by the Collection Fusion process which is also a distributed retrieval process. Ranking on the data is based on how probable is the data to give the required information to the user. It picks up the best resources and ranks it. The resources which are selected are combined in a list and given to the user.Data FusionFigure 9 Data Fusion ArchitectureAbove figure give the architectural reference of the Data Fusion.L0 The user information which the sensors and the sources process is given to the fusion algorithmL1 The machine process of converting L0 output and it feeds it to the L1 processL2 L3 The human process of manually marking the retrieved data as relevant or notL4 choice management layer adds to the sources and the L2/L3 layer providing th necessary feedbackL6 The mission management helps to add context to the user query. This is added to the resource management layerL5 Once all the processing through the layers is complete the data is given back to the user.This is fusion mechanism which relies on the common data sets which are retrieved by search engines. The initial ranks are combined into a list. The analysis of the actual ranks of the documents retrieved is calculated. The links of the documents with the highest relevancy score is selected. ComboSum is one of the techniques used to normalize the scores. The scores produced are incomparable as different search engines run on different algorithms.Examples of Meta-Search EnginesDeepPeepFigure 10 DeepPeep SnippetAbove figure is a snippet for DeepPeet. Public web pages are indexes and expanded by search engine. It does so by search through various public databases. Traditional search engines cannot index by DeepPeep. It also looks to find urls into deep web.IncyWincyFigure 11 Incywincy snippetAbove image is Incywincy snippet. It uses many search engines, combines and filters the results. It crawlers more than 200 million pages. It uses its unique algorithm for relevance tests.CompletePlanetFigure 12 Complete PlanetAbove figure is snippet of Complete Planet. It indexes and crawlers many search engines and special databases which are not public. It is one of the main search engines of the deep web.Other Deep Web search engines include Intute, Infomine, Scirus, Turbo 10.The above figure is the Turbo 10 is a meta search engine snippet. It helps to engineer a universal reference for Deep Net. It refers to topic specific search engines. Google and Yahoo like crawler-based search engines do not use these files. Turbo 10 also lets you add more engines to the collection on which your search query is executed. 7ProsThe positive thing about turbo10 is that it has the ability to connect and keep on connection to online databases in bulk. This provides connecting to thousand engines in a fully automated capability which is scalable to further connect to another thousand.Turbo10 also finds content on deep net. Deep net encompasses multiple databases covering wide range of topics such as business, colleges, government teams. These are not accessible to traditional web engines and google. Traditional search engines are helpful for indexing static pages.Thus a rich experience is provided as we can tap into multiple databases across such a large range of domains, empowering the user.Turbo provides the option of having ten search engines to search from.Unlike other search engine like AskJeeves, that faces drawback of segregating information into different boxes, turbo streamlines the process by proving the result in weighted listing. This makes Turbo10 the ultimate search engine of search engine.Consider the fact that say four search engines are selected, turbo10 does the work of selecting additional six search engines as it deems as best fit.The result set would be based on either relevance or speed. The decision is independent of the search independent of search engine selected.The above reduction of search result occurs by usage of clusters in box on left-hand side. The benefit of having clusters a few time, desired result set is obtained quicker as against advanced searches or logical expressionTurbo10 provides a rising search paradigm. For a given page, there are ten result lists on a page. These have arrows. Thought this may also be unpopular, this generates hundred results for a given search.The average case being thirty to forty result set being provided as three or four pages are returned. . Google gives lots of options. Unlike Google, Turbo10 simply limits the search results and provides limited result setThe main idea is that with Turbo 10, ability to choose search enigne is provided. Whereas say for a particular item through google, we need to keep searching and may find the same at pat a latter page say page fortyHow is the ability provided to select ten search engines?The task is completed via the web. A collection name is assigned to the ten search engines, mapped to your email.CapabilityTurbo10 has a vision to emerge as a leader for search enginePlan to use amazon based recommendation algorithms for personalised searching. This would include personalized profile search, providing bookmarking feature an ecosystem to be setup wherein the users profile would interact with other users. Additional browsing options would be generated for users. Revenue model would tap into sponsored web links. These would be flagged licensing of the product to corporates.ConsTturbo10 faces the drawback of being highly cluttered. Google has lots of white spaces. The search results are not cramped up. Turbo10 is not visually appealing. It has purple colour screen that may not be appealing to allMany times the returned result set is cluttered. Sometimes the result set gets jumbled affec ting result set. Additionally, there is no consideration for punctuation marks or logical expressions. Does not have ability to cache information. Due to high demand and expectation, knocked offline.Intense emulation with google.The project is based on dark net ecosystem comprising of file sharing mechanisms and meta search engine. Dark net has continuously using evolving technology such as TOR, OnionShare. These help enable provide benefits of being anonymous protecting their identity. Similarly, the website to gets to preserve their anonymity. Dark net serves as a platform of communication to mediate, take decision and share information. These are used by different agencies including government and non-traditional activities such as whistleblowing.Products that are not listed in traditional websites too are purchasable on such platforms.File Sharing approaches save bandwidth by helping sharing information efficiently. Easier to back up information. Fault tolerance is enhanced th rough decentralized approach part of peer to peer networking. Comparatively there is ease of maintenance over traditional files sharing system.Meta search engines help to get streamlined results from various search engines which helps to improve relevancy of searches.The benefits of dark net also include cross border payments, ensure complete privacy of transmitter and receiver. There is application in cryptocurrency, digital trading, eliminating middlemen. Set up of hassle free payments independent of weekend or holidays. Benefit of decentralization is control over your content that you want to share. Other advantages include establish net neutrality which means that internet can be used for all and not monopolized.Hence there is a large scope for expansion in dark net space and technology is playing a pivotal role in enhancing its adoption.1 https//www.linkedin.com/pulse/internet-deep-web-dark-net-firas-saras2 https//sysinfotools.com/blog/peer-to-peer-file-sharing/3 http//www.spi roprojects.com/blog/cat-view-more.php?blogname=What-is-file-sharing-system?id=2624 https//www.howtogeek.com/275875/what-is-the-dark-web/5 https//darkwebnews.com/anonymity/some-ways-to-share-files-anonymously/6 Johan Andersson Gabriel Ledung, Darknet file sharing application of a private peer-to-peer distributed file system concept7 http//techdiction.blogspot.com/2007/01/turbo-10-search-deep-net.htmlURL Uniform Resource LocatorTORThe Onion RouterUDPUser Datagram ProtocolDTHdistributed hash tableWWWWorldwide web