Differences

This shows you the differences between two versions of the page.

--- qortal_data_network_qdn [11/01/2022 13:27] – gfactor
+++ qortal_data_network_qdn [11/02/2023 18:19] (current) – crowetic
@@ Line 1: / Line 1: @@
 ====== Qortal Data Network (QDN) ======
-{{:qortal_official_logo_transparent_.png?600|}}
+{{:qortal_official_logo_transparent_.png?400|}}
-{{::54582a96-0c0c-4f3a-b6b1-8b0ed0a9b9bc.jpeg?600|}}
-**Qortal Data Network (QDN) is LIVE! Although there is a data size limit of 500MB as we begin building this into a scalable solution to rebuild the Internet as we know it. Bear with us!**
-QDN is a 100% FREE solution to rebuilding the Internet in a decentralized fashion. Instead of paying third-parties for a domain name, simply register a name on Qortal and own that domain for LIFE. Then you’ll be able to host your own website with whatever data you’d like - free from any censorship or control by Big Tech. You simply host your data on your Qortal node, and a tiny hash is registered on Qortal’s chain to log your ownership. Your data is NOT hosted on Qortal’s chain, as that would not be a scalable solution for rebuilding the Internet ;-). The data is hosted on your hardware, and the website link is found in Qortal’s UI under the ‘Websites’ section. The person wanting to browse your site will then click the link, and download the data to their node in order to view it. Yes, it’s kind of a strange concept but is our first phase of rolling out a truly decentralized solution. The data is encrypted during transfer, and can be either 1) deleted after viewing or 2) supported as a peer. The latter option means you can choose to ‘Follow’ that website and run a redundant copy of that data - which keeps the website live if the original host’s node goes offline for whatever reason. This also increases the speed of accessing the data by the rest of the network. By working hand-in-hand, we can support each other in building a web of data that is unhackable and self-empowering.
+=====QDN - Qortal Data Network=====
-Now what if you come across ‘illicit’ content during your QDN browsing? Simply click the red x next to the website you’ve viewed and delete the data. Of course this raises a lot of concerns for the typical user and we completely understand. Please keep in mind that we are building a set of layers to Qortal’s infrastructure and perfecting this will take time. The plan is to develop a White List that users can choose to opt into, which we can work together to build and essentially block out the illicit content. We will also have another White List for kids, like a YouTube for kids, so parents can feel confident that their child is safely browsing the Qortal Network. True decentralization means we have truly no control over what others post, but we reserve control over our own nodes and the data we choose to access.
+QDN (Qortal Data Network) is a **Distributed/Decentralized/Encrypted/Chunked Content Distribution Network** that is **secured by the Qortal Blockchain.**
-Data can be relayed through our nodes, in order for other nodes to communicate. All of this data is end-to-end encrypted, and is also chunked so the data is not relayed on a single node. The data being relayed is only on your node long enough to conduct the transfer, and is deleted afterwards automatically. Users can choose to opt out of relay mode if their lack of understanding with this matter raises too much concern. Getting data around the network is difficult and requires more fine tuning- even with relay mode enabled (which helps the network). To disable relay mode, add this setting to your json settings:
+QDN is **unlike any other data network** in the world, and has many unique features only found on Qortal.
-<code>"relayModeEnabled": false</code>
+The Qortal Data Network provides the ability to:
-QDN is accessible to everyone and does not cost anything. The only cost is registering a name (current name registration fee can be found on the [[Qortal Status Updates]] page.
+  * Host Websites
+  * Host Applications
+  * Communications
+  * Host Public Data
+  * Host Private Data
+  * Build nearly anything you can imagine
-We came up with a method of data storage that could be better than using IPFS. We have a working prototype already which works in a similar way to IPFS but we hope will be a lot more user friendly. It’s only for static sites at the moment (HTML, JS, CSS, images, other static assets) so you would need to either build it directly as a static site, or take a static copy of an existing site using a tool such as httrack, SiteSucker, Simply Static, etc.
+The Qortal Data Network is completely **encrypted and chunked** - This means that the data transmitted from node-to-node is completely encrypted, and unable to be 'seen' at all by any 'outside methods'. For example:
-Here are two examples of sites hosted on a Qortal data node:
+An internet service provider wishes to 'spy' on the data their clients are accessing. So they put packet capture on, and take that data to analyze. On the traditional internet they can see what sites they are accessing, they can see if they are downloading a torrent, they can see basically everything that is going on within the network's captured data.
-http://node1.qortal.uk:12393/site/QortalDemo
+On QDN, however, they would not be able to see any of this, the only information they would be able to obtain, would be that there is a connection from one computer to another, sending data (that has no meaning whatsoever) back and forth. They would be completely unable to determine what this data is, who published it, where it came from initially, etc.
-http://node1.qortal.uk:12393/site/QortalWikiDemo
+This makes it essentially completely unnecessary to run things like a 'VPN' while utilizing Qortal. In fact, utilizing a VPN on the Qortal Network will do nothing but potentially cause connection issues by adding an additional 'hop' between you and the nodes you are connected to. There is literally only ONE arguable 'benefit' from utilizing a VPN in combination with Qortal, and that is that your IP address will not be 'yours', it will be the IP of the VPN server. (But in many cases this isn't necessarily a good thing, as many providers block connections from VPN servers that are known, and this could cause further issues.)
-Here is an example of a data node that acts as a gateway for traditional users - allowing them to access QDN sites without having to run a node: https://qortal.link/crowetic
+Ultimately, QDN is MUCH more secure than any other data network, and attempting to add 'additional security' to it, is wholly unnecessary.
-These are older static copies of the Qortal website and wiki so they may not be up to date FYI.
-At first, it will be more simple, since we will only be doing simple static websites, and we can convert wordpress sites to be hosted on Qortal's infrastructure. So if you view a website, as a data node, you will become a peer for it, unless you state that you don't want to in settings. You don't have to worry about someone uploading something to another namespace, as it simply isn't possible. The only people who can modify data on a registered name's space, is the person who owns the name. Qortal's blockchain has the registered names, just like it does now for the registered names we use for 'usernames'. Registered names will just have a data location tied to them as well; 'name storage'. A Qortal data node can act as a 'proxy' from a traditional domain. So you can host a traditional domain on Qortal as well.
+(this page is in process of being updated, and completely re-written, more information coming soon.)
-Data hosting will essentially be a clone of Qortal, without the synchronizer and a different chain that will keep track of the data hashes. The data transmission will be done like a torrent. The more peers you have the more redundant copies you will have. You will obtain peers by obtaining 'followers' or by another data node viewing your public data (unless that other node decides they do not want to be a host of your data.) Data nodes will have 100% control over what they host and what they don't but by default, if you're a data node, and you access another data node's public data, you will become a peer for it.
+====QDN Details====
-Public data is only encrypted in terms of the account that put the data there, the encryption only controls the MODIFICATION of that data. Private data, on the other hand, will be fully encrypted. We’re not entirely certain yet on how we'll decide which private data is held by other nodes. Potentially we will just have an option to allow private data to be stored as a duplicate on your node. You’ll be rewarded for that data from the fees the person putting the data up will pay for the data to be stored/duplicated on the network. So as a data node you will have the option to allow private data to be stored, then as the person wanting that data stored/duplicated, you will pay a fee for however many duplicate copies you would like to store. The fee then goes to the data node that ends up storing that duplicate copy.
+QDN functions in a similar fashion to that of the 'torrent' network, however, with many additional features and functions that set it apart.
-Here’s the code being worked on so far for the data hosting: https://github.com/CalDescent14/qortal-data
+QDN data is by default **encrypted, and 'chunked'** - this means that the data is by default unable to be viewed by anyone, even if they stole the hard drive from your computer, unless you had specifically clicked that data and viewed it, and did not delete the temporary copy.
-===== Notes From CalDescent =====
+====QDN Website Hosting====
-The Qortal data nodes currently work very similarly to an AWS S3 bucket (but in a decentralized way). They essentially give you an interface to add, retrieve and update static "resources"; each resource is really just a folder structure. You could very easily take these files into a centralized app via the data node APIs and then process/execute them in whatever way you like.
+With a single registered name that you **own forever and never have to pay any further fees on** you can host your website on the Qortal Data Network.
-I am also in the process of building several "interfaces" built on top of the static files. The current planned ones are:
+  * Register a name
+  * Create your website in 'local static' format
+  * Zip the website
+  * Publish
-  -Static site serving (mostly finished)
+That's it! It is possible to convert existing websites to 'static' websites, and publish them as well. (Note, certain websites do not convert as nicely as others. Some do not convert well at all.) There are tools like 'httrack' that can accomplish the static conversion for you.
-  -Git repositories
-  -Blog
-The idea with these interfaces is they allow static file structures to be served and written to by the data nodes. For static sites, it's mostly just a case of serving them over a webserver, but for blogs it will be a custom plugin that renders the blog and backs the data off to JSON files - using it as the database essentially.
+Once you have a website that is fully accessible from your local computer (folders on your machine) and everything works locally, simply zip that folder with the site files in it, and publish on QDN. From the 'websites' plugin in the Qortal UI, 'publish website' and fill out the metadata fields. It is literally that simple.
-For git repositories, I am planning on using an open source Java git client such as Gitblit and back its data off to the static files. That way you can add your data node as a remote and pull/push to it.
+Once you have published the site ONCE, as long as at least one other node has accessed it or follows your name, the site will be there indefinitely. No concerns over hacks, no concerns over data loss, and no continual hosting or domain fees involved at all.
-I hadn't thought of doing a "static website builder", but that is a really great idea. If we can allow people to build sites within Qortal using an open source project, that would save a lot of work, and would be a really nice feature. One potential issue with Jekyll is that it's not java based, so it would be great if we could find a native java alternative. Otherwise it's complicated to bundle it with our app.
-In terms of dynamic content, I haven't got too far into the details of that since we can achieve most of our plans using staticly hosted files plus "smart interfaces" on top - such as git or the website builder. One day we may be able to have, for instance, a PHP interface in which we allow the files to be executed by PHP. But it won't be too soon as the sandboxing would be non trivial - it may need virtual machines or possibly Docker to ensure the execution is fully isolated and locked down.
-Databases are even more difficult. I will probably start by making a MySQL interface that backs off its data to the static files. But I worry that it will be impractical for a lot of use cases, because each INSERT or UPDATE would require a new transaction to be published on chain. So it would only work for infrequent updates, such as a basic website.
-We are limiting the amount of transactions / resource updates using a proof of work nonce, so that will stop someone from being able to spam the chain constantly, which will have the effect of limiting to only "low throughput" use cases. The data nodes handle partial updates already, so it only updates the changes instead of re-uploading the entire file structure every time.
-Here's how "decentralized LAMP" could work in the future:
-  -Add a MySQL resource and publish the initial database schema and data. This would create a transaction on chain and sync the associated data around the network.
-  -Add a PHP resource and publish your initial PHP files.
-  -Inside your PHP database class you would point it to the Qortal MySQL instance. This would essentially map to http://localhost:12393/mysql/MyRegisteredName
-  -Deploy the PHP resource which would add a transaction and sync its data around the network.
-  -Other users could then run the PHP app (and database) using by visiting `http://localhost:12392/MyRegisteredName`
-  -The MySQL resource would have permissions set so that it can be updated by all (or a certain group of people, or just the original creator - depending on the use case).
-  -Everyone's nodes would watch for local changes to the MySQL repo, and submit a transaction automatically when needed.
-Obviously the approach above has major drawbacks for serious / high throughput use cases, and also ones that need to store sensitive data. But it would work in projects that update their data infrequently (e.g. WordPress), so it could be a fairly straightforward way to get started. Most of the foundations are in place for this already.
-The current plan is to include a java-based version of httrack in the data nodes, so that it can automatically convert a URL and add the static copy to the data chain. After that, we'll have the ability to include a bot in the data nodes that watches a given URL for changes and then automatically creates a static copy and publishes any differences to the chain each time. I'm not sure whether this feature will make it into v1 of data nodes; it will most depend on whether there is a java-based static site downloader already available. If not we'll have to create our own, which will take a while.
-The storage nodes will have their own API endpoints to access and update data. They'll also have the API endpoints that we're already used to, so that you can access blocks and transactions on the data chain.
-===== More Notes From CalDescent =====
-The way public data hosting works at the moment is by using confirmable transactions (so that there is always an on-chain record of updates to each website/service), but rather than a transaction fee, we require a proof of work nonce to be calculated and included with the transaction. The difficulty depends on the size of the files being added to the network. We use a similar PoW nonce for Q-chat transactions.
-Then, in order for the data itself to replicate, the creator must have viewers of the data, or followers of their account. This allows good data to propagate, whereas data that has no viewers or followers wouldn't be taking up space on people's nodes. It also allows node owners to control exactly what data they are hosting, by following only those creators that they want to support (similar to torrents).
-The idea behind using a nonce rather than a transaction fee removes the cost barrier so encourages people to use the system. But since the uploader has to perform some difficult proof of work calculations each time, it prevents someone from easily spamming the chain.
-Also, the data hashes are being written to a completely separate chain for scalability reasons (it doesn't fill up the blocks on the main chain and also allows us to have multiple independent data chains when we need to scale up). In v1, there will be no way to spend QORT on the data chain, so there is no scope for transaction fees. But in subsequent versions we will add a "bridge" to move QORT between chains. At this point, we can add features such as allowing uploaders to pay extra for more copies of their data to be stored, and we can reward data nodes for the data they are hosting.
-All of the above relates to "public data" only. We will eventually have support for "private data" too, and this would work completely differently. It would likely be a case of paying QORT to host it, which would be paid out to the participating data nodes. But that feature won't be in place for a while.
-Name registration doubles as a “domain name” for websites and other data hosted by that account (in v3.0). Currently owned domain names will not be possible to transfer registration to your account on Qortal. They are completely separate and there won't be any way of linking them. Traditional domain names rely on "domain authorities" such as ICANN, which are incompatible with Qortal's system and ethos. Best thing to do is just register your existing names with Qortal. There's currently no established convention in terms of domain extensions in Qortal. I expect most names won't have an extension, but it's possible one such as .qort will emerge organically in the future. You should be able to register .com extensions etc within the system if you prefer them over names with no extensions.
-You can absolutely access it via a separate browser. There are a couple of options:
-. Using the API on a public/gateway node such as http://node1.qortal.uk:12393/site/QortalDemo
-. Using the “domain mapping” feature with existing DNS - e.g http://www.reqorder.org/ - this site is on the data testnet
-When setting up domain mapping: you will need to add the domain name to your node(s) and then point the domain at them via DNS. You can have multiple nodes behind a single domain for better redundancy.
-This “gateway” feature is now developed, so we can already support gateways to the Qortal network via an existing domain. E.g: <code>https://gateway.qortal.org/RegisteredName</code>
-or
-<code>https://qortalweb.com/RegisteredName</code>
-etc.
-Basically anyone who is willing to can provide access to the entire Qortal network of websites via a domain they own. They just have to enable the gateway service in their node’s settings (and point their domain at the node using DNS). You can always connect the traditional domain name to your Qortal site if you want to. It will just rely on centralised components, unlike using a registered name within Qortal.
-===== We Use PoW For Data Uploads =====
-We use PoW for data uploads to reduce the ability for people to spam the chain with data. For most transaction types in Qortal, we use a 0.001 QORT transaction fee. For QDN we decided to remove the fee and replace it with a "nonce", which essentially means a certain amount of computational resources need to be used when publishing data, to slow down requests and reduce spam. We use this approach for Q-Chat messages (most significantly when the account has a zero balance) and also for some existing on-chain transaction types, such as the "MESSAGE" transactions that are used in the trade portal (to allow people to buy QORT without having any QORT to create the trade portal Buy transaction).
-The main reason for doing this for QDN was because we were planning on putting data hashes on a sidechain that wouldn't support transaction fees. Due to time constraints we ended up putting data hashes on the main chain (there's still lots of capacity there) but once we need to scale up we will need to return to the sidechain idea, and when we do, it will make it easier for both devs and users if we are using a proof of work nonce rather than a transaction fee.

Qortal Project

User Tools

Site Tools

Differences

Page Tools