diff --git a/_sources/adapter/index.md.txt b/_sources/adapter/index.md.txt deleted file mode 100644 index eb8922d..0000000 --- a/_sources/adapter/index.md.txt +++ /dev/null @@ -1,11 +0,0 @@ -# Adapter - -Interfaces to other protocols and formats - -- Files - - hdf5 - - json - - csv - - mat -- HTTP -- S3 \ No newline at end of file diff --git a/_sources/codecs/hdf5.md.txt b/_sources/codecs/hdf5.md.txt new file mode 100644 index 0000000..5f552a9 --- /dev/null +++ b/_sources/codecs/hdf5.md.txt @@ -0,0 +1,15 @@ +# HDF5 + +We are starting with hdf5 because our initial test case is the [NWB](https://www.nwb.org/) format for neurophysiology data. This is a challenging initial test case because the data is heterogeneous, large, and the specification is written in an idiosyncratic format specification. + +HDF has three primary types of objects: +- Groups +- Datasets - contain the raw values in the file +- Attributes - metadata about groups or datasets. + +Datasets have additional properties: +- Datatypes: binary representation of the data +- Dataspaces: Layout of individual data elements +- Properties: Additional information about the representation of the dataset, eg. chunked or contiguous + +These map naturally onto triplets, where each group or dataset is a subject, attributes are properties and objects. \ No newline at end of file diff --git a/_sources/codecs/index.md.txt b/_sources/codecs/index.md.txt new file mode 100644 index 0000000..b14ab3e --- /dev/null +++ b/_sources/codecs/index.md.txt @@ -0,0 +1,20 @@ +# Codecs + +Interfaces to file formats + +We want to support three kinds of interaction with files: + +- **References** - treat files like abstract binary with some metadata indicating file type and a hash tree for the file +- **Introspection** - Export some metadata from the file that indicates components of the file along with byte ranges. We want to be able to know what is inside the file without downloading it, but we keep the file separate as an out of protocol entity. +- **Ingestion** - Export the metadata and the data contained within the file to triples. We also store some translation between the original binary file and the resulting triple through translation schema that allows us to update our triples if the files change and otherwise keep a strong link to the source, but otherwise enable forking/querying/etc. as if the data does not have an underlying file. + +This is a challenging design balance, where we don't want clients to need to implement a large number of codecs for different files - so they can fall back to the reference strategy as needed - but we also want people to be able to interact and import their files without needing to abandon longstanding practices or other infrastructure they might already have for using/creating them. + +```{toctree} +hdf5 +``` + +- Files + - json + - csv + - mat \ No newline at end of file diff --git a/_sources/comparison/data/datalad.md.txt b/_sources/comparison/data/datalad.md.txt new file mode 100644 index 0000000..aeae965 --- /dev/null +++ b/_sources/comparison/data/datalad.md.txt @@ -0,0 +1,15 @@ +```{index} DataLad +``` +# DataLad + +DataLad is a tool for managing datasets! It is built on top of {index}`git` and {index}`git-annex ` for storage, and is capable of integrating with external hosting providers (through git annex) + +It has a number of interesting extensions that we can learn from +- [crawler](https://docs.datalad.org/projects/crawler/en/latest/) allows you to archive web pages +- [OSF Remote](https://github.com/datalad/datalad-osf/blob/main/datalad_osf/annex_remote.py) - example of an extension for interacting with OSF + + +## References + +- DataLad repositories: https://github.com/datalad +- Docs: https://docs.datalad.org/en/latest/ \ No newline at end of file diff --git a/_sources/comparison/data/index.md.txt b/_sources/comparison/data/index.md.txt index b7ac6f9..a51e3c7 100644 --- a/_sources/comparison/data/index.md.txt +++ b/_sources/comparison/data/index.md.txt @@ -4,6 +4,7 @@ :caption: Data Structures :maxdepth: 1 -eris +datalad dmc +eris ``` \ No newline at end of file diff --git a/_sources/comparison/ld/hdt.md.txt b/_sources/comparison/ld/hdt.md.txt new file mode 100644 index 0000000..a5c8555 --- /dev/null +++ b/_sources/comparison/ld/hdt.md.txt @@ -0,0 +1,123 @@ +```{index} Linked Data; HDT +``` +(hdt)= +# HDT + +Like [Linked Data Fragments](ld_fragments), [HDT](https://www.rdfhdt.org/) is a transport and query format for linked data triples. + +It is a compressed format that preserves headers to enable query and browsing without decompression. + +## Format + +It has [three components](https://www.rdfhdt.org/technical-specification/): + +{attribution="https://www.rdfhdt.org/technical-specification/"} +> - **Header:** The Header holds metadata describing an HDT semantic dataset using plain RDF. It acts as an entry point for the consumer, who can have an initial idea of key properties of the content even before retrieving the whole dataset. +> - **Dictionary:** The Dictionary is a catalog comprising all the different terms used in the dataset, such as URIs, literals and blank nodes. A unique identifier (ID) is assigned to each term, enabling triples to be represented as tuples of three IDs, which reference their respective subject/predicate/object term from the dictionary. This is a first step toward compression, since it avoids long terms to be repeated again and again. Moreover, similar strings are now stored together inside the dictionary, fact that can be exploited to improve compression even more. +> - **Triples:** As stated before, the RDF triples can now be seen as tuples of three IDs. Therefore, the Triples section models the graph of relationships among the dataset terms. By understanding the typical properties of RDF graphs, we can come up with more efficient ways of representing this information, both to reduce the overall size, but also to provide efficient search/traversal operations. + +### Header + +A header contains + +- At least one resource of type `hdt:Dataset`, which has + - Publication metadata - Where and when the dataset was published + - Statistical metadata - Number of triples, number of terms, etc. + - Format metadata - Encoding of dataset, which must have + - `hdt:dictionary` + - `hdt:triples` + - Additional metadata - uh idk anything? + + +````{dropdown} HDT Header Example +```turtle +@prefix void: . +@prefix dc: . +@prefix foaf: . +@prefix hdt: . +@prefix xsd: . +@prefix rdfs: . +@prefix rdf: . +@prefix swp: . + + + a hdt:Dataset ; + a void:Dataset ; + hdt:publicationInformation :publication ; + hdt:statisticalInformation :statistics ; + hdt:formatInformation :format ; + hdt:additionalInformation :additional ; + void:triples "431440396" ; + void:properties "57986" ; + void:distinctSubjects "24791728" ; + void:distinctObjects "108927201" . + +:publication dc:issued "2012-11-23T23:17:50+0000" ; + dc:license ; + dc:publisher [ a foaf:Organization ; + foaf:homepage ] ; + dc:source ; + dc:title "DBpediaEN" ; + void:sparqlEndpoint . + +:statistics hdt:originalSize "110630364018" ; + hdt:hdtSize "3082795954" . + +:format hdt:dictionary :dictionary ; + hdt:triplesBitmap :triples . + +:dictionary dc:format hdt:dictionaryFour ; + hdt:dictionaryNamespaces [hdt:namespace [hdt:prefixLabel "dbpedia" ; + hdt:prefixURI "http://dbpedia.org/resource/"]] ; + hdt:dictionarynumSharedSubjectObject "22762644" ; + hdt:dictionarysizeStrings "1026354060" ; + hdt:dictionaryBlockSize "8" . + +:triples dc:format hdt:triplesBitmap ; + hdt:triplesOrder "SPO" ; + hdt:triplesnumTriples "431440396" . + +:additional swp:signature "AZ8QWE..." ; + swp:signatureMethod "DSA" . +``` +```` + +### Dictionary + +The dictionary replaces all terms in the dataset with short, unique IDs to make the dataset more compressible. Oddly, rather than being a simple lookup table, it splits the dictionary into four sections: a "shared" section that includes subjects and objects, and predicates are separated. Terms are lexicographically ordered and [front coded](https://en.wikipedia.org/wiki/Incremental_encoding) to additionally aid compression. + +Separating encoding information into a header dictionary is a straightforwardly good idea, and an argument for distributing linked data in 'packetized' forms rather than as a bunch of raw triples, as we do here. + +### Triples + +Triples are encoded as a tree, where each subject forms a root, with each predicate as children, and likewise for objects. Since the dictionary is ordered such that the subjects are the lowest IDs, it is possible to use an implicit representation of each subject (ie. subjects are not encoded). The predicate and object layers are each encoded with two parallel bit streams: Each predicate or object entry has one `Sp` entry for its dictionary ID, and one `Bp` "bitsequence" entry which is `1` if the entry is the first child of its parent and `0` otherwise. + +## Querying + +The dictionary being uncompressed allows for the dataset to be indexed at a vocabulary level - it is possible to eg. 'find all datasets that use this set of terms,' as well as slightly more refined queries like 'find datasets that use this term as both subject and object.' + +Lookup is fast for subject-based queries, but predicate and object queries are slower because of the bitmap triple encoding. + + + + + + +## Lessons + +First, there are good strategies here for practical compression and serialization of RDF triples! + +The most interesting thing for p2p-ld here is the header: we are also interested in making it possible to do restricted queries and indexing over containers of triples without needing to necessarily query, download, or unpack the entire dataset. The primary focus here is compression, which has add-on benefits like faster query performance because the dataset can be held in memory. We would instead like to focus on exposing hashed tree fragments that can encapsulate query logic - eg. a given RDF resource that might indicate the metadata for a type of experiment would be hashed as a tree, and queries can discover it by querying for the root or any of its child hashes. So we will take the ideas re: using the dictionary encoding without necessarily adopting HDT wholesale. + +The bitmap encoding is also interesting, as according to their tests it outperforms other similar compression schemes and I/O times. We will keep this in mind as a potential serialization format for raw triple data. + +The idea of including publication data in the header seems obvious, but according to the authors later work that is not necessarily the case in RDF world {cite}`polleresMoreDecentralizedVision2020`. Since p2p-ld is built explicitly around making identity and origin a more central component of linked data, we will further investigate using the {index}`VOID vocabulary ` - https://www.w3.org/TR/void/ + + + + +## References + +- [HDT Homepage](https://www.rdfhdt.org/) +- Original Paper: {cite}`fernandezBinaryRDFRepresentation2013` +- Later contextualization: {cite}`polleresMoreDecentralizedVision2020` \ No newline at end of file diff --git a/_sources/comparison/ld/index.md.txt b/_sources/comparison/ld/index.md.txt index 0baedda..722fd27 100644 --- a/_sources/comparison/ld/index.md.txt +++ b/_sources/comparison/ld/index.md.txt @@ -7,6 +7,14 @@ rdf solid ld_fragments +hdt ld_platform nanopubs -``` \ No newline at end of file +``` + +Linked data was born to be p2p. Many of the [initial, lofty visions](https://jon-e.net/surveillance-graphs/#semantic-web-priesthoods) of the [semantic web](https://jon-e.net/infrastructure/#linked-data-has-an-ambivalent-history-of-thought-regarding-the-l) are only possible with p2p systems - fluid, languagelike ontologies, portable personal data, truly decentralized information structuring on the web and so on {cite}`saundersSurveillanceGraphs2023,saundersDecentralizedInfrastructureNeuro2022`. That's one of the central goals of this project --- as might be obvious from its placeholder name: p2p-ld. + +Don't just take my word for it tho: + +{attribution="A more decentralized vision for Linked Data. Polleres et al. (2020)"} +> So, where does this leave us? We have seen a lot of resources being put into publishing Linked Data, but yet a publicly widely visible “killer app” is still missing. The reason for this, in the opinion and experiences of the authors, lies all to often in the frustrating experiences when trying to actually use Linked Data for building actual applications. Many attempts and projects end up still using a centralized warehousing approach, integrating a handful of data sets directly from their raw data sources, rather than being able to leverage their “lifted” Linked Data versions: the use and benefits of RDF and Linked Data over conventional databases and warehouses technologies, where more trained people are available, remain questionable. {cite}`polleresMoreDecentralizedVision2020` \ No newline at end of file diff --git a/_sources/comparison/ld/rdf.md.txt b/_sources/comparison/ld/rdf.md.txt index 110398c..076c824 100644 --- a/_sources/comparison/ld/rdf.md.txt +++ b/_sources/comparison/ld/rdf.md.txt @@ -12,7 +12,9 @@ We depart from that vision, instead favoring radical vernacularism {cite}`saunde ## RDF And Friends -RDF has a lot of formats and +```{important} +Return here re: RDF canonicalization and IPFS https://github.com/multiformats/multicodec/pull/261 +``` ```{index} JSON-LD ``` @@ -23,11 +25,7 @@ RDF has a lot of formats and ## Challenges -### Tabular and Array Data - -```{important} -See https://www.cs.ox.ac.uk/isg/challenges/sem-tab/ -``` +### Ordered Data The edges from a node in a graph are unordered, which makes array and tabular data difficult to work with in RDF! @@ -44,16 +42,16 @@ eg. one would express `MyList` which contains the `Friends` `["Arnold", "Bob", " :MyList :Friends :list1 . :list1 - rdf:first :Amy ; - rdf:rest :list2 . + rdf:first :Amy ; + rdf:rest :list2 . :list2 - rdf:first :Bob ; - rdf:rest :list3 . + rdf:first :Bob ; + rdf:rest :list3 . :list3 - rdf:first :Carly ; - rdf:rest rdf:nil . + rdf:first :Carly ; + rdf:rest rdf:nil . ``` And thankfully turtle has a shorthand, which isn't so bad: @@ -63,33 +61,33 @@ And thankfully turtle has a shorthand, which isn't so bad: @prefix : . :MyList - :Friends ( - :Amy - :Bob - :Carly - ). + :Friends ( + :Amy + :Bob + :Carly + ). ``` Both of these correspond to the triplet graph: ```{mermaid} flowchart LR - MyList - list1 - list2 - list3 - nil - Amy - Bob - Carly - - MyList -->|Friends| list1 - list1 -->|rest| list2 - list2 -->|rest| list3 - list3 -->|rest| nil - list1 -->|first| Amy - list2 -->|first| Bob - list3 -->|first| Carly + MyList + list1 + list2 + list3 + nil + Amy + Bob + Carly + + MyList -->|Friends| list1 + list1 -->|rest| list2 + list2 -->|rest| list3 + list3 -->|rest| nil + list1 -->|first| Amy + list2 -->|first| Bob + list3 -->|first| Carly ``` Which is not great. @@ -152,6 +150,315 @@ which can be expanded recursively to [mimic arrays](https://www.w3.org/TR/json-l ```` ````` +### Tabular Data + +As an overbrief summary, converting data from tables to RDF needs a schema mapping: + +- Columns to Properties +- +- Column names in source table to symbolic names used within the conversion schema +- datatype (for representation in concrete RDF syntax) +- + + +According to the [Tabular Data to RDF](https://www.w3.org/TR/csv2rdf/) recommendation, one would convert the following table (encoded as `csv`): + +```{csv-table} +countryCode,latitude,longitude,name +AD,42.5,1.6,Andorra +AE,23.4,53.8,"United Arab Emirates" +AF,33.9,67.7,Afghanistan +``` + +Into one of two "minimal" or "standard" formats of RDF: + +`````{tab-set} +````{tab-item} Minimal mode +```turtle +@base . + +:8228a149-8efe-448d-b15f-8abf92e7bd17 + <#countryCode> "AD" ; + <#latitude> "42.5" ; + <#longitude> "1.6" ; + <#name> "Andorra" . + +:ec59dcfc-872a-4144-822b-9ad5e2c6149c + <#countryCode> "AE" ; + <#latitude> "23.4" ; + <#longitude> "53.8" ; + <#name> "United Arab Emirates" . + +:e8f2e8e9-3d02-4bf5-b4f1-4794ba5b52c9 + <#countryCode> "AF" ; + <#latitude> "33.9" ; + <#longitude> "67.7" ; + <#name> "Afghanistan" . +``` +```` +````{tab-item} Standard mode +```turtle +@base . +@prefix csvw: . +@prefix xsd: . + +:d4f8e548-9601-4e41-aadb-09a8bce32625 a csvw:TableGroup ; + csvw:table [ a csvw:Table ; + csvw:url ; + csvw:row [ a csvw:Row ; + csvw:rownum "1"^^xsd:integer ; + csvw:url <#row=2> ; + csvw:describes :8228a149-8efe-448d-b15f-8abf92e7bd17 + ], [ a csvw:Row ; + csvw:rownum "2"^^xsd:integer ; + csvw:url <#row=3> ; + csvw:describes :ec59dcfc-872a-4144-822b-9ad5e2c6149c + ], [ a csvw:Row ; + csvw:rownum "3"^^xsd:integer ; + csvw:url <#row=4> ; + csvw:describes :e8f2e8e9-3d02-4bf5-b4f1-4794ba5b52c9 + ] + ] . + +:8228a149-8efe-448d-b15f-8abf92e7bd17 + <#countryCode> "AD" ; + <#latitude> "42.5" ; + <#longitude> "1.6" ; + <#name> "Andorra" . + +:ec59dcfc-872a-4144-822b-9ad5e2c6149c + <#countryCode> "AE" ; + <#latitude> "23.4" ; + <#longitude> "53.8" ; + <#name> "United Arab Emirates" . + +:e8f2e8e9-3d02-4bf5-b4f1-4794ba5b52c9 + <#countryCode> "AF" ; + <#latitude> "33.9" ; + <#longitude> "67.7" ; + <#name> "Afghanistan" . +``` +```` +````` + +The recommendation also covers more complex situations. These make use of a JSON schema that handles mapping between the CSV data and RDF. + +By default, each row of a table describes a single RDF resource, and each column has a single property (so each cell is a triple). + +For example this table of concerts: + +```{csv-table} +Name, Start Date, Location Name, Location Address, Ticket Url +B.B. King,2014-04-12T19:30,"Lupo’s Heartbreak Hotel","79 Washington St., Providence, RI",https://www.etix.com/ticket/1771656 +B.B. King,2014-04-13T20:00,"Lynn Auditorium","Lynn, MA, 01901",http://frontgatetickets.com/venue.php?id=11766 +``` + +Needs to be mapped to 3 separate resources with 7 properties. The values are not transformed, just grouped in different places under different resources. Notice how in the standard mode the `csvw:describes`{l=turtle} entry can have three objects. The turtle is surprisingly humane. + +The JSON schema describes five concrete triples that carry the data from the CSV, and five `virtual` triples that give the resources types and link them together. Abstractions over table iterators take the form of `"#event-{_row}"` to create a resource `<#event-1>`, `<#event-2>`, etc. for each row. + +`````{tab-set} +````{tab-item} Minimal mode +```turtle +@base . +@prefix schema: . +@prefix xsd: . + +<#event-1> a schema:MusicEvent ; + schema:name "B.B. King" ; + schema:startDate "2014-04-12T19:30:00"^^xsd:dateTime ; + schema:location <#place-1> ; + schema:offers <#offer-1> . + +<#place-1> a schema:Place ; + schema:name "Lupo’s Heartbreak Hotel" ; + schema:address "79 Washington St., Providence, RI" . + +<#offer-1> a schema:Offer ; + schema:url "https://www.etix.com/ticket/1771656"^^xsd:anyURI . + +<#event-2> a schema:MusicEvent ; + schema:name "B.B. King" ; + schema:startDate "2014-04-13T20:00:00"^^xsd:dateTime ; + schema:location <#place-2> ; + schema:offers <#offer-2> . + +<#place-2> a schema:Place ; + schema:name "Lynn Auditorium" ; + schema:address "Lynn, MA, 01901" . + +<#offer-2> a schema:Offer ; + schema:url "http://frontgatetickets.com/venue.php?id=11766"^^xsd:anyURI . +``` +```` +````{tab-item} Standard mode +```turtle +@base . +@prefix csvw: . +@prefix schema: . +@prefix xsd: . + +:95cc7970-ce99-44b0-900c-e2c2c028bbd3 a csvw:TableGroup ; + csvw:table [ a csvw:Table ; + csvw:url ; + csvw:row [ a csvw:Row ; + csvw:rownum 1 ; + csvw:url <#row=2> ; + csvw:describes <#event-1>, <#place-1>, <#offer-1> + ], [ a csvw:Row ; + csvw:rownum 2 ; + csvw:url <#row=3> ; + csvw:describes <#event-2>, <#place-2>, <#offer-2> + ] + ] . + +<#event-1> a schema:MusicEvent ; + schema:name "B.B. King" ; + schema:startDate "2014-04-12T19:30:00"^^xsd:dateTime ; + schema:location <#place-1> ; + schema:offers <#offer-1> . + +<#place-1> a schema:Place ; + schema:name "Lupo’s Heartbreak Hotel" ; + schema:address "79 Washington St., Providence, RI" . + +<#offer-1> a schema:Offer ; + schema:url "https://www.etix.com/ticket/1771656"^^xsd:anyURI . + +<#event-2> a schema:MusicEvent ; + schema:name "B.B. King" ; + schema:startDate "2014-04-13T20:00:00"^^xsd:dateTime ; + schema:location <#place-2> ; + schema:offers <#offer-2> . + +<#place-2> a schema:Place ; + schema:name "Lynn Auditorium" ; + schema:address "Lynn, MA, 01901" . + +<#offer-2> a schema:Offer ; + schema:url "http://frontgatetickets.com/venue.php?id=11766"^^xsd:anyURI . +``` +```` +````{tab-item} JSON Schema +```json +{ + "@context": ["http://www.w3.org/ns/csvw", {"@language": "en"}], + "url": "events-listing.csv", + "dialect": {"trim": true}, + "tableSchema": { + "columns": [{ + "name": "name", + "titles": "Name", + "aboutUrl": "#event-{_row}", + "propertyUrl": "schema:name" + }, { + "name": "start_date", + "titles": "Start Date", + "datatype": { + "base": "datetime", + "format": "yyyy-MM-ddTHH:mm" + }, + "aboutUrl": "#event-{_row}", + "propertyUrl": "schema:startDate" + }, { + "name": "location_name", + "titles": "Location Name", + "aboutUrl": "#place-{_row}", + "propertyUrl": "schema:name" + }, { + "name": "location_address", + "titles": "Location Address", + "aboutUrl": "#place-{_row}", + "propertyUrl": "schema:address" + }, { + "name": "ticket_url", + "titles": "Ticket Url", + "datatype": "anyURI", + "aboutUrl": "#offer-{_row}", + "propertyUrl": "schema:url" + }, { + "name": "type_event", + "virtual": true, + "aboutUrl": "#event-{_row}", + "propertyUrl": "rdf:type", + "valueUrl": "schema:MusicEvent" + }, { + "name": "type_place", + "virtual": true, + "aboutUrl": "#place-{_row}", + "propertyUrl": "rdf:type", + "valueUrl": "schema:Place" + }, { + "name": "type_offer", + "virtual": true, + "aboutUrl": "#offer-{_row}", + "propertyUrl": "rdf:type", + "valueUrl": "schema:Offer" + }, { + "name": "location", + "virtual": true, + "aboutUrl": "#event-{_row}", + "propertyUrl": "schema:location", + "valueUrl": "#place-{_row}" + }, { + "name": "offers", + "virtual": true, + "aboutUrl": "#event-{_row}", + "propertyUrl": "schema:offers", + "valueUrl": "#offer-{_row}" + }] + } +} +``` +```` +````` + +One could imagine how this might generalize into multidimensional array data, but that immediately becomes pretty ridiculous - a better strategy in all cases that I can think of would be to just provide metadata about the array like the encoding, the sizes, types, etc. of their axes and indices and then link to the array. + +I'll just leave this example of encoding the pixels in one RGB video frame as a joke. + +```turtle +@prefix vid: . +@prefix xsd: . +@prefix rdf: . + +:myVideo a vid:VideoGroup ; + vid:video [ a vid:Video ; + vid:url ; + vid:frame [ a vid:Frame ; + vid:framenum 1 ; + vid:url <#frame=1> ; + vid:describes <#frame-1> ; + ], [ a vid:Frame ; + vid:framenum 2 ; + vid:url <#frame=2> ; + vid:describes <#frame-2> ; + ] + ] . + +<#frame-1> a vid:VideoFrame ; + vid:timestamp "2023-06-29T12:00:00"^^xsd:dateTime ; + vid:bitDepth 8 ; + vid:width 1920 ; + vid:height 1080 ; + vid:channels <#red-1>, <#green-1>, <#blue-1> ; + +<#red-1> a vid:VideoChannel ; + :pixel-1 a vid:pixelValue ; + rdf:first 0 ; + rdf:rest :pixel-2 . + + :pixel-2 a vid:pixelValue ; + rdf:first 46 ; + rdf:rest :pixel-3 . + + # ... + + :pixel-2073600 a vid:pixelValue ; + rdf:first 57 ; + rdf:rest rdf:nil . +``` + ### Naming - All names have to be global. Relative names must resolve to a global name via contexts/prefixes. The alternative is blank nodes, which are treated as equivalent in eg. graph merges. Probably here enters pattern matching or whatever those things are called. @@ -162,6 +469,8 @@ which can be expanded recursively to [mimic arrays](https://www.w3.org/TR/json-l - [RDF 1.1 Primer](https://www.w3.org/TR/rdf11-primer/) - W3C Recommendation on generating RDF from tabular data: {cite}`tandyGeneratingRDFTabular2015` + - Tabular data model: https://www.w3.org/TR/2015/REC-tabular-data-model-20151217/#parsing + - Metadata model: https://www.w3.org/TR/2015/REC-tabular-metadata-20151217/ - {index}`JSON Schema` in RDF: {cite}`charpenayJSONSchemaRDF2023` - [Turtle](https://www.w3.org/TR/rdf12-turtle/) - [N-ary relations in RDF](https://www.w3.org/TR/swbp-n-aryRelations/) @@ -173,11 +482,19 @@ which can be expanded recursively to [mimic arrays](https://www.w3.org/TR/json-l - [rdf-canonize-native](https://github.com/digitalbazaar/rdf-canonize-native) - [biolink-model](https://github.com/biolink/biolink-model) for a nice example of generating multiple schema formats from a .yaml file. - [linkml](https://linkml.io/) - modeling language for linked data {cite}`moxonLinkedDataModeling2021` - - Multidimensional arrays in linkml https://linkml.io/linkml/howtos/multidimensional-arrays.html + - Multidimensional arrays in linkml https://linkml.io/linkml/howtos/multidimensional-arrays.html - [oaklib](https://incatools.github.io/ontology-access-kit/index.html) - python package for managing ontologies - [rdflib](https://github.com/RDFLib/rdflib) - maybe the canonical python rdf library +- [csv2rdf](https://github.com/Swirrl/csv2rdf/) ### See Also - [HYDRA vocabulary](https://www.hydra-cg.com/spec/latest/core/) - Linked Data plus REST -- [CORAL](https://github.com/jmchandonia/CORAL) \ No newline at end of file +- [CORAL](https://github.com/jmchandonia/CORAL) +- [SEMTAB](https://www.cs.ox.ac.uk/isg/challenges/sem-tab/) - competition for mapping tabular data to RDF +- [SciSPARQL](https://www.ceur-ws.org/Vol-1272/paper_22.pdf) - an extension of SPARQL to include arrays. + +### Example Datasets + +- [RDF Data Dumps](https://www.w3.org/wiki/DataSetRDFDumps) +- [bio2rdf](https://download.bio2rdf.org) \ No newline at end of file diff --git a/_sources/comparison/p2p/bittorrent.md.txt b/_sources/comparison/p2p/bittorrent.md.txt index edbc924..8d393b6 100644 --- a/_sources/comparison/p2p/bittorrent.md.txt +++ b/_sources/comparison/p2p/bittorrent.md.txt @@ -76,6 +76,38 @@ Though not explicitly in the protocol spec, two prominent design decisions are w - **Peer Selection:** Which peers should I spent finite bandwidth uploading to? BitTorrent uses a variety of **Choke** algorithms that reward peers that reciprocate bandwidth. Choke algorithms are typically some variant of a 'tit-for-tat' strategy, although rarely the strict bitwise tit-for-tat favored by later blockchain systems and others that require a peer to upload an equivalent amount to what they have downloaded before they are given any additional pieces. Contrast this with [{index}`BitSwap`](#BitSwap) from IPFS. It is by *not* perfectly optimizing peer selection that BitTorrent is better capable of using more of its available network resources. - **Piece Selection:** Which pieces should be uploaded/requested first? BitTorrent uses a **Rarest First** strategy, where a peer keeps track of the number of copies of each piece present in the swarm, and preferentially seeds the rarest pieces. This keeps the swarm healthy, rewarding keeping and sharing complete copies of files. This is in contrast to, eg. [SWARM](#SWARM) which explicitly rewards hosting and sharing the most in-demand pieces. +```{index} Web Seeds +``` +## Web Seeds + +One thing we want to mimic from bittorrent is the ability to use traditional web servers as additional peers, or to treat them as ["WebSeeds"](http://bittorrent.org/beps/bep_0019.html)[^BEP17] + +HTTP servers allow you to specify a byte range to resume a download, but don't like the downloading client connecting hundreds of times to download the same file, jumping between pieces. To accomodate that, BEP 19 changes piece selection accordingly: + +When downloading from bittorrent peers, we modify the "rarest first" algorithm such that for pieces with similar rareness we +- Select pieces from smaller "gaps" in between completed blocks +- Select pieces closer to the end of the gap +- After 50% of the torrent is completed, for some random subset of pieces, ignore rarest first and fill in small gaps. + +When downloading from HTTP servers +- Start from some random location in the file (to avoid every peer having the same pieces at the start of the file) +- When partially completed, select the next longest gap between completed pieces + +For multi-file torrents +- Prefer bittorrent downloads for small files that are less than a piece size + +We can consider {index}`libtorrent `'s implementation as a reference implementation. +- Libtorrent chooses pieces by [starting by assuming the client has all files and eliminating pieces for files we don't have](https://github.com/arvidn/libtorrent/blob/c2012b084c6654d681720ea0693d87a48bc95b14/src/web_peer_connection.cpp#L165-L171). +- On requesting a piece, it [checks for resume data](https://github.com/arvidn/libtorrent/blob/c2012b084c6654d681720ea0693d87a48bc95b14/src/web_peer_connection.cpp#L368-L394) if we have already partially downloaded it before, and modifies the start and length of the piece request +- It then [constructs an HTTP GET request](https://github.com/arvidn/libtorrent/blob/c2012b084c6654d681720ea0693d87a48bc95b14/src/web_peer_connection.cpp#L423-L442), using the [Range](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Range) header to select some subsection of the file. +- When we [receive data](https://github.com/arvidn/libtorrent/blob/c2012b084c6654d681720ea0693d87a48bc95b14/src/web_peer_connection.cpp#L778) from the server, we wait until we receive the full header, then we parse the body of the response. If the size is different than what we expected, we disconnect from the server. Otherwise, we iterate through any chunks and store them. +- If the pieces received from the web seed [fail the hash check](https://github.com/arvidn/libtorrent/blob/c2012b084c6654d681720ea0693d87a48bc95b14/src/web_peer_connection.cpp#L578-L584), we mark the peer as not having the file, which bans it in the case of a single file torrent, but allows us to check whether the other files on the server have been changed. + + + + + + ## Lessons @@ -88,7 +120,6 @@ Though not explicitly in the protocol spec, two prominent design decisions are w - `.torrent` files make for a very **low barrier to entry** and are extremely **portable.** They also operate over the existing idioms of files and folders, rather than creating their own filesystem abstraction. - Explicit peer and piece selection algorithms are left out of the protocol specification, allowing individual implementations to experiment with what works. This makes it possible to exploit the protocol by refusing to seed ever, but this rarely occurs in practice, as people are not the complete assholes imagined in worst-case scenarios of scarcity. Indeed even the most selfish peers have the intrinsic incentive to upload, as by aggressively seeding the pieces that a leeching peer already has, the other peers in the swarm are less likely to "waste" the bandwidth of the seeders and more bandwidth can be allocated to pieces that the leecher doesn't already have. - ### Adapt - **Metadata**. Currently all torrent metadata is contained within the tracker, so while it is possible to restore all the files that were indexed by a downed tracker, it is very difficult to restore all the metadata at a torrent level and above, eg. the organization of specific torrents into hierarchical categories that allow one to search for an artist, all the albums they have produced, all the versions of that album in different file formats, and so on. @@ -99,13 +130,22 @@ Though not explicitly in the protocol spec, two prominent design decisions are w 2. Maintain the possibility for loose anonymity where peers can share files without needing a large and well-connected social system to share files to them 3. Avoid significant performance penalties from guarantees of strong network-level anonymity like Tor. - **Trackers** are a good idea, even if they could use some updating. It is good to have an explicit entrypoint specified with a distributed, social mechanism rather than prespecified as a hardcoded entry point. It is a good idea to make a clear space for social curation of information, rather than something that is intrinsically bound to a torrent at the time of uploading. We update the notion of trackers with [Peer Federations](#Peer-Federations). +- **Web Seeds** + - Torrent files handle single and multi-file torrents similarly, with the file structure in the info-dict. We can instead explicitly follow the lead of Bittorrent v2.0 and have per-file hash trees and URL references, avoiding some of the ambiguity in the web seed implementation that [requires us to do some manual path traversal](https://github.com/arvidn/libtorrent/blob/c2012b084c6654d681720ea0693d87a48bc95b14/src/web_peer_connection.cpp#L101-L121) + - We want to be able to integrate with existing servers and services, so we want to be able to find files by both the URL of the original file (if that is its "canonical" location) and its hash. Rather than adding a web seed as an additional source of a torrent file, we can treat it as one of the additional identifiers for the given container. This adds an additional argument in favor of nested containers as the unit of exchange. Eg. A data repository might have a single URL for a dataset that has multiple files within it, and the individual files might not have unique URLs (eg. the file picker generates a .zip file on the fly). A peer might want to bundle together multiple files from different locations. So it should be possible for each container to have multiple names, and when another peers requests a file by eg. a URL we can look within our containers for a match. This also allows handling files that might be uploaded in multiple places + - We want to store the [Last-Modified](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Last-Modified) data when importing a file from a web seed so that we can handle version changes in a given file without giving up on the web source entirely. When the `Last-Modified` is updated, we get the new file, re-hash it, and update the relevant file container if it has been changed. Otherwise we just store the new `Last-Modified` ## References - Bittorrent Protocol Specification (BEP 3): http://www.bittorrent.org/beps/bep_0003.html - Bittorrent v2 (BEP 52): http://www.bittorrent.org/beps/bep_0052.html - Magnet Links (BEP 9): http://www.bittorrent.org/beps/bep_0009.html +- WebSeeds (BEP 19): http://bittorrent.org/beps/bep_0019.html - More on BitTorrent and incentives - {cite}`cohenIncentivesBuildRobustness2003` +- Notes about writing a bittorrent client from the GetRight author, particularly re: DHT: https://www.getright.com/torrentdev.html +- Nice example of implementing a very minimal bittorrent client in Python: https://markuseliasson.se/article/bittorrent-in-python/ -[^announcelist]: Or, properly, in the `announce-list` per ([BEP 12](http://www.bittorrent.org/beps/bep_0012.html)) \ No newline at end of file +[^announcelist]: Or, properly, in the `announce-list` per ([BEP 12](http://www.bittorrent.org/beps/bep_0012.html)) + +[^BEP17]: There is a parallel [BEP 17](https://www.bittorrent.org/beps/bep_0017.html) that allows modified HTTP servers to more directly seed, but since it requires changes to existing servers we are less concerned with it. \ No newline at end of file diff --git a/_sources/comparison/p2p/ipfs.md.txt b/_sources/comparison/p2p/ipfs.md.txt index dd80a02..971c819 100644 --- a/_sources/comparison/p2p/ipfs.md.txt +++ b/_sources/comparison/p2p/ipfs.md.txt @@ -45,3 +45,11 @@ If IPFS is {index}`BitTorrent` + {index}`git`, and {index}`ActivityPub` is {inde ## Differences - Not permanent storage! Identities retain custody and control over objects in the network. + + +## References + +- [IPFS-LD](https://github.com/ipfs/ipfs/issues/36) +- Discussions on gateways and {index}`Web Seed `-like things in IPFS: + - https://github.com/ipfs/kubo/issues/8234 + - \ No newline at end of file diff --git a/_sources/index.md.txt b/_sources/index.md.txt index d9ac97e..1ab8d49 100644 --- a/_sources/index.md.txt +++ b/_sources/index.md.txt @@ -41,9 +41,9 @@ evolvability :hidden: triplets -adapter/index +codecs/index translation/index -``` +``` ```{toctree} :caption: Drafting @@ -59,7 +59,7 @@ sketchpad genindex references -``` +``` Indices and tables ================== diff --git a/_sources/translation.md.txt b/_sources/translation.md.txt deleted file mode 100644 index 0f23512..0000000 --- a/_sources/translation.md.txt +++ /dev/null @@ -1,8 +0,0 @@ -# Translation - -A toolkit for writing translations between formats and schemas! - -## See also - -- https://linkml.io/schema-automator/introduction.html#generalization-from-instance-data -- https://apps.islab.ntua.gr/d2rml/tr/d2rml/ \ No newline at end of file diff --git a/_sources/translation/index.md.txt b/_sources/translation/index.md.txt index b102829..e0dfec1 100644 --- a/_sources/translation/index.md.txt +++ b/_sources/translation/index.md.txt @@ -2,3 +2,7 @@ Translation/import of existing schema/formats. +## See also + +- https://linkml.io/schema-automator/introduction.html#generalization-from-instance-data +- https://apps.islab.ntua.gr/d2rml/tr/d2rml/ diff --git a/_static/pygments.css b/_static/pygments.css index d9a83a7..08f8ec6 100644 --- a/_static/pygments.css +++ b/_static/pygments.css @@ -83,173 +83,175 @@ .highlight .il { color: #0000cf; font-weight: bold } /* Literal.Number.Integer.Long */ @media not print { body[data-theme="dark"] .highlight pre { line-height: 125%; } -body[data-theme="dark"] .highlight td.linenos .normal { color: #aaaaaa; background-color: transparent; padding-left: 5px; padding-right: 5px; } -body[data-theme="dark"] .highlight span.linenos { color: #aaaaaa; background-color: transparent; padding-left: 5px; padding-right: 5px; } -body[data-theme="dark"] .highlight td.linenos .special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; } -body[data-theme="dark"] .highlight span.linenos.special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; } -body[data-theme="dark"] .highlight .hll { background-color: #404040 } -body[data-theme="dark"] .highlight { background: #202020; color: #d0d0d0 } -body[data-theme="dark"] .highlight .c { color: #ababab; font-style: italic } /* Comment */ -body[data-theme="dark"] .highlight .err { color: #a61717; background-color: #e3d2d2 } /* Error */ -body[data-theme="dark"] .highlight .esc { color: #d0d0d0 } /* Escape */ -body[data-theme="dark"] .highlight .g { color: #d0d0d0 } /* Generic */ -body[data-theme="dark"] .highlight .k { color: #6ebf26; font-weight: bold } /* Keyword */ -body[data-theme="dark"] .highlight .l { color: #d0d0d0 } /* Literal */ -body[data-theme="dark"] .highlight .n { color: #d0d0d0 } /* Name */ -body[data-theme="dark"] .highlight .o { color: #d0d0d0 } /* Operator */ -body[data-theme="dark"] .highlight .x { color: #d0d0d0 } /* Other */ -body[data-theme="dark"] .highlight .p { color: #d0d0d0 } /* Punctuation */ -body[data-theme="dark"] .highlight .ch { color: #ababab; font-style: italic } /* Comment.Hashbang */ -body[data-theme="dark"] .highlight .cm { color: #ababab; font-style: italic } /* Comment.Multiline */ -body[data-theme="dark"] .highlight .cp { color: #ff3a3a; font-weight: bold } /* Comment.Preproc */ -body[data-theme="dark"] .highlight .cpf { color: #ababab; font-style: italic } /* Comment.PreprocFile */ -body[data-theme="dark"] .highlight .c1 { color: #ababab; font-style: italic } /* Comment.Single */ -body[data-theme="dark"] .highlight .cs { color: #e50808; font-weight: bold; background-color: #520000 } /* Comment.Special */ -body[data-theme="dark"] .highlight .gd { color: #d22323 } /* Generic.Deleted */ -body[data-theme="dark"] .highlight .ge { color: #d0d0d0; font-style: italic } /* Generic.Emph */ -body[data-theme="dark"] .highlight .gr { color: #d22323 } /* Generic.Error */ -body[data-theme="dark"] .highlight .gh { color: #ffffff; font-weight: bold } /* Generic.Heading */ -body[data-theme="dark"] .highlight .gi { color: #589819 } /* Generic.Inserted */ -body[data-theme="dark"] .highlight .go { color: #cccccc } /* Generic.Output */ -body[data-theme="dark"] .highlight .gp { color: #aaaaaa } /* Generic.Prompt */ -body[data-theme="dark"] .highlight .gs { color: #d0d0d0; font-weight: bold } /* Generic.Strong */ -body[data-theme="dark"] .highlight .gu { color: #ffffff; text-decoration: underline } /* Generic.Subheading */ -body[data-theme="dark"] .highlight .gt { color: #d22323 } /* Generic.Traceback */ -body[data-theme="dark"] .highlight .kc { color: #6ebf26; font-weight: bold } /* Keyword.Constant */ -body[data-theme="dark"] .highlight .kd { color: #6ebf26; font-weight: bold } /* Keyword.Declaration */ -body[data-theme="dark"] .highlight .kn { color: #6ebf26; font-weight: bold } /* Keyword.Namespace */ -body[data-theme="dark"] .highlight .kp { color: #6ebf26 } /* Keyword.Pseudo */ -body[data-theme="dark"] .highlight .kr { color: #6ebf26; font-weight: bold } /* Keyword.Reserved */ -body[data-theme="dark"] .highlight .kt { color: #6ebf26; font-weight: bold } /* Keyword.Type */ -body[data-theme="dark"] .highlight .ld { color: #d0d0d0 } /* Literal.Date */ -body[data-theme="dark"] .highlight .m { color: #51b2fd } /* Literal.Number */ -body[data-theme="dark"] .highlight .s { color: #ed9d13 } /* Literal.String */ -body[data-theme="dark"] .highlight .na { color: #bbbbbb } /* Name.Attribute */ -body[data-theme="dark"] .highlight .nb { color: #2fbccd } /* Name.Builtin */ -body[data-theme="dark"] .highlight .nc { color: #71adff; text-decoration: underline } /* Name.Class */ -body[data-theme="dark"] .highlight .no { color: #40ffff } /* Name.Constant */ -body[data-theme="dark"] .highlight .nd { color: #ffa500 } /* Name.Decorator */ -body[data-theme="dark"] .highlight .ni { color: #d0d0d0 } /* Name.Entity */ -body[data-theme="dark"] .highlight .ne { color: #bbbbbb } /* Name.Exception */ -body[data-theme="dark"] .highlight .nf { color: #71adff } /* Name.Function */ -body[data-theme="dark"] .highlight .nl { color: #d0d0d0 } /* Name.Label */ -body[data-theme="dark"] .highlight .nn { color: #71adff; text-decoration: underline } /* Name.Namespace */ -body[data-theme="dark"] .highlight .nx { color: #d0d0d0 } /* Name.Other */ -body[data-theme="dark"] .highlight .py { color: #d0d0d0 } /* Name.Property */ -body[data-theme="dark"] .highlight .nt { color: #6ebf26; font-weight: bold } /* Name.Tag */ -body[data-theme="dark"] .highlight .nv { color: #40ffff } /* Name.Variable */ -body[data-theme="dark"] .highlight .ow { color: #6ebf26; font-weight: bold } /* Operator.Word */ -body[data-theme="dark"] .highlight .pm { color: #d0d0d0 } /* Punctuation.Marker */ -body[data-theme="dark"] .highlight .w { color: #666666 } /* Text.Whitespace */ -body[data-theme="dark"] .highlight .mb { color: #51b2fd } /* Literal.Number.Bin */ -body[data-theme="dark"] .highlight .mf { color: #51b2fd } /* Literal.Number.Float */ -body[data-theme="dark"] .highlight .mh { color: #51b2fd } /* Literal.Number.Hex */ -body[data-theme="dark"] .highlight .mi { color: #51b2fd } /* Literal.Number.Integer */ -body[data-theme="dark"] .highlight .mo { color: #51b2fd } /* Literal.Number.Oct */ -body[data-theme="dark"] .highlight .sa { color: #ed9d13 } /* Literal.String.Affix */ -body[data-theme="dark"] .highlight .sb { color: #ed9d13 } /* Literal.String.Backtick */ -body[data-theme="dark"] .highlight .sc { color: #ed9d13 } /* Literal.String.Char */ -body[data-theme="dark"] .highlight .dl { color: #ed9d13 } /* Literal.String.Delimiter */ -body[data-theme="dark"] .highlight .sd { color: #ed9d13 } /* Literal.String.Doc */ -body[data-theme="dark"] .highlight .s2 { color: #ed9d13 } /* Literal.String.Double */ -body[data-theme="dark"] .highlight .se { color: #ed9d13 } /* Literal.String.Escape */ -body[data-theme="dark"] .highlight .sh { color: #ed9d13 } /* Literal.String.Heredoc */ -body[data-theme="dark"] .highlight .si { color: #ed9d13 } /* Literal.String.Interpol */ -body[data-theme="dark"] .highlight .sx { color: #ffa500 } /* Literal.String.Other */ -body[data-theme="dark"] .highlight .sr { color: #ed9d13 } /* Literal.String.Regex */ -body[data-theme="dark"] .highlight .s1 { color: #ed9d13 } /* Literal.String.Single */ -body[data-theme="dark"] .highlight .ss { color: #ed9d13 } /* Literal.String.Symbol */ -body[data-theme="dark"] .highlight .bp { color: #2fbccd } /* Name.Builtin.Pseudo */ -body[data-theme="dark"] .highlight .fm { color: #71adff } /* Name.Function.Magic */ -body[data-theme="dark"] .highlight .vc { color: #40ffff } /* Name.Variable.Class */ -body[data-theme="dark"] .highlight .vg { color: #40ffff } /* Name.Variable.Global */ -body[data-theme="dark"] .highlight .vi { color: #40ffff } /* Name.Variable.Instance */ -body[data-theme="dark"] .highlight .vm { color: #40ffff } /* Name.Variable.Magic */ -body[data-theme="dark"] .highlight .il { color: #51b2fd } /* Literal.Number.Integer.Long */ +body[data-theme="dark"] .highlight td.linenos .normal { color: #6e7681; background-color: #0d1117; padding-left: 5px; padding-right: 5px; } +body[data-theme="dark"] .highlight span.linenos { color: #6e7681; background-color: #0d1117; padding-left: 5px; padding-right: 5px; } +body[data-theme="dark"] .highlight td.linenos .special { color: #e6edf3; background-color: #6e7681; padding-left: 5px; padding-right: 5px; } +body[data-theme="dark"] .highlight span.linenos.special { color: #e6edf3; background-color: #6e7681; padding-left: 5px; padding-right: 5px; } +body[data-theme="dark"] .highlight .hll { background-color: #6e7681 } +body[data-theme="dark"] .highlight { background: #0d1117; color: #e6edf3 } +body[data-theme="dark"] .highlight .c { color: #8b949e; font-style: italic } /* Comment */ +body[data-theme="dark"] .highlight .err { color: #f85149 } /* Error */ +body[data-theme="dark"] .highlight .esc { color: #e6edf3 } /* Escape */ +body[data-theme="dark"] .highlight .g { color: #e6edf3 } /* Generic */ +body[data-theme="dark"] .highlight .k { color: #ff7b72 } /* Keyword */ +body[data-theme="dark"] .highlight .l { color: #a5d6ff } /* Literal */ +body[data-theme="dark"] .highlight .n { color: #e6edf3 } /* Name */ +body[data-theme="dark"] .highlight .o { color: #ff7b72; font-weight: bold } /* Operator */ +body[data-theme="dark"] .highlight .x { color: #e6edf3 } /* Other */ +body[data-theme="dark"] .highlight .p { color: #e6edf3 } /* Punctuation */ +body[data-theme="dark"] .highlight .ch { color: #8b949e; font-style: italic } /* Comment.Hashbang */ +body[data-theme="dark"] .highlight .cm { color: #8b949e; font-style: italic } /* Comment.Multiline */ +body[data-theme="dark"] .highlight .cp { color: #8b949e; font-weight: bold; font-style: italic } /* Comment.Preproc */ +body[data-theme="dark"] .highlight .cpf { color: #8b949e; font-style: italic } /* Comment.PreprocFile */ +body[data-theme="dark"] .highlight .c1 { color: #8b949e; font-style: italic } /* Comment.Single */ +body[data-theme="dark"] .highlight .cs { color: #8b949e; font-weight: bold; font-style: italic } /* Comment.Special */ +body[data-theme="dark"] .highlight .gd { color: #ffa198; background-color: #490202 } /* Generic.Deleted */ +body[data-theme="dark"] .highlight .ge { color: #e6edf3; font-style: italic } /* Generic.Emph */ +body[data-theme="dark"] .highlight .gr { color: #ffa198 } /* Generic.Error */ +body[data-theme="dark"] .highlight .gh { color: #79c0ff; font-weight: bold } /* Generic.Heading */ +body[data-theme="dark"] .highlight .gi { color: #56d364; background-color: #0f5323 } /* Generic.Inserted */ +body[data-theme="dark"] .highlight .go { color: #8b949e } /* Generic.Output */ +body[data-theme="dark"] .highlight .gp { color: #8b949e } /* Generic.Prompt */ +body[data-theme="dark"] .highlight .gs { color: #e6edf3; font-weight: bold } /* Generic.Strong */ +body[data-theme="dark"] .highlight .gu { color: #79c0ff } /* Generic.Subheading */ +body[data-theme="dark"] .highlight .gt { color: #ff7b72 } /* Generic.Traceback */ +body[data-theme="dark"] .highlight .g-Underline { color: #e6edf3; text-decoration: underline } /* Generic.Underline */ +body[data-theme="dark"] .highlight .kc { color: #79c0ff } /* Keyword.Constant */ +body[data-theme="dark"] .highlight .kd { color: #ff7b72 } /* Keyword.Declaration */ +body[data-theme="dark"] .highlight .kn { color: #ff7b72 } /* Keyword.Namespace */ +body[data-theme="dark"] .highlight .kp { color: #79c0ff } /* Keyword.Pseudo */ +body[data-theme="dark"] .highlight .kr { color: #ff7b72 } /* Keyword.Reserved */ +body[data-theme="dark"] .highlight .kt { color: #ff7b72 } /* Keyword.Type */ +body[data-theme="dark"] .highlight .ld { color: #79c0ff } /* Literal.Date */ +body[data-theme="dark"] .highlight .m { color: #a5d6ff } /* Literal.Number */ +body[data-theme="dark"] .highlight .s { color: #a5d6ff } /* Literal.String */ +body[data-theme="dark"] .highlight .na { color: #e6edf3 } /* Name.Attribute */ +body[data-theme="dark"] .highlight .nb { color: #e6edf3 } /* Name.Builtin */ +body[data-theme="dark"] .highlight .nc { color: #f0883e; font-weight: bold } /* Name.Class */ +body[data-theme="dark"] .highlight .no { color: #79c0ff; font-weight: bold } /* Name.Constant */ +body[data-theme="dark"] .highlight .nd { color: #d2a8ff; font-weight: bold } /* Name.Decorator */ +body[data-theme="dark"] .highlight .ni { color: #ffa657 } /* Name.Entity */ +body[data-theme="dark"] .highlight .ne { color: #f0883e; font-weight: bold } /* Name.Exception */ +body[data-theme="dark"] .highlight .nf { color: #d2a8ff; font-weight: bold } /* Name.Function */ +body[data-theme="dark"] .highlight .nl { color: #79c0ff; font-weight: bold } /* Name.Label */ +body[data-theme="dark"] .highlight .nn { color: #ff7b72 } /* Name.Namespace */ +body[data-theme="dark"] .highlight .nx { color: #e6edf3 } /* Name.Other */ +body[data-theme="dark"] .highlight .py { color: #79c0ff } /* Name.Property */ +body[data-theme="dark"] .highlight .nt { color: #7ee787 } /* Name.Tag */ +body[data-theme="dark"] .highlight .nv { color: #79c0ff } /* Name.Variable */ +body[data-theme="dark"] .highlight .ow { color: #ff7b72; font-weight: bold } /* Operator.Word */ +body[data-theme="dark"] .highlight .pm { color: #e6edf3 } /* Punctuation.Marker */ +body[data-theme="dark"] .highlight .w { color: #6e7681 } /* Text.Whitespace */ +body[data-theme="dark"] .highlight .mb { color: #a5d6ff } /* Literal.Number.Bin */ +body[data-theme="dark"] .highlight .mf { color: #a5d6ff } /* Literal.Number.Float */ +body[data-theme="dark"] .highlight .mh { color: #a5d6ff } /* Literal.Number.Hex */ +body[data-theme="dark"] .highlight .mi { color: #a5d6ff } /* Literal.Number.Integer */ +body[data-theme="dark"] .highlight .mo { color: #a5d6ff } /* Literal.Number.Oct */ +body[data-theme="dark"] .highlight .sa { color: #79c0ff } /* Literal.String.Affix */ +body[data-theme="dark"] .highlight .sb { color: #a5d6ff } /* Literal.String.Backtick */ +body[data-theme="dark"] .highlight .sc { color: #a5d6ff } /* Literal.String.Char */ +body[data-theme="dark"] .highlight .dl { color: #79c0ff } /* Literal.String.Delimiter */ +body[data-theme="dark"] .highlight .sd { color: #a5d6ff } /* Literal.String.Doc */ +body[data-theme="dark"] .highlight .s2 { color: #a5d6ff } /* Literal.String.Double */ +body[data-theme="dark"] .highlight .se { color: #79c0ff } /* Literal.String.Escape */ +body[data-theme="dark"] .highlight .sh { color: #79c0ff } /* Literal.String.Heredoc */ +body[data-theme="dark"] .highlight .si { color: #a5d6ff } /* Literal.String.Interpol */ +body[data-theme="dark"] .highlight .sx { color: #a5d6ff } /* Literal.String.Other */ +body[data-theme="dark"] .highlight .sr { color: #79c0ff } /* Literal.String.Regex */ +body[data-theme="dark"] .highlight .s1 { color: #a5d6ff } /* Literal.String.Single */ +body[data-theme="dark"] .highlight .ss { color: #a5d6ff } /* Literal.String.Symbol */ +body[data-theme="dark"] .highlight .bp { color: #e6edf3 } /* Name.Builtin.Pseudo */ +body[data-theme="dark"] .highlight .fm { color: #d2a8ff; font-weight: bold } /* Name.Function.Magic */ +body[data-theme="dark"] .highlight .vc { color: #79c0ff } /* Name.Variable.Class */ +body[data-theme="dark"] .highlight .vg { color: #79c0ff } /* Name.Variable.Global */ +body[data-theme="dark"] .highlight .vi { color: #79c0ff } /* Name.Variable.Instance */ +body[data-theme="dark"] .highlight .vm { color: #79c0ff } /* Name.Variable.Magic */ +body[data-theme="dark"] .highlight .il { color: #a5d6ff } /* Literal.Number.Integer.Long */ @media (prefers-color-scheme: dark) { body:not([data-theme="light"]) .highlight pre { line-height: 125%; } -body:not([data-theme="light"]) .highlight td.linenos .normal { color: #aaaaaa; background-color: transparent; padding-left: 5px; padding-right: 5px; } -body:not([data-theme="light"]) .highlight span.linenos { color: #aaaaaa; background-color: transparent; padding-left: 5px; padding-right: 5px; } -body:not([data-theme="light"]) .highlight td.linenos .special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; } -body:not([data-theme="light"]) .highlight span.linenos.special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; } -body:not([data-theme="light"]) .highlight .hll { background-color: #404040 } -body:not([data-theme="light"]) .highlight { background: #202020; color: #d0d0d0 } -body:not([data-theme="light"]) .highlight .c { color: #ababab; font-style: italic } /* Comment */ -body:not([data-theme="light"]) .highlight .err { color: #a61717; background-color: #e3d2d2 } /* Error */ -body:not([data-theme="light"]) .highlight .esc { color: #d0d0d0 } /* Escape */ -body:not([data-theme="light"]) .highlight .g { color: #d0d0d0 } /* Generic */ -body:not([data-theme="light"]) .highlight .k { color: #6ebf26; font-weight: bold } /* Keyword */ -body:not([data-theme="light"]) .highlight .l { color: #d0d0d0 } /* Literal */ -body:not([data-theme="light"]) .highlight .n { color: #d0d0d0 } /* Name */ -body:not([data-theme="light"]) .highlight .o { color: #d0d0d0 } /* Operator */ -body:not([data-theme="light"]) .highlight .x { color: #d0d0d0 } /* Other */ -body:not([data-theme="light"]) .highlight .p { color: #d0d0d0 } /* Punctuation */ -body:not([data-theme="light"]) .highlight .ch { color: #ababab; font-style: italic } /* Comment.Hashbang */ -body:not([data-theme="light"]) .highlight .cm { color: #ababab; font-style: italic } /* Comment.Multiline */ -body:not([data-theme="light"]) .highlight .cp { color: #ff3a3a; font-weight: bold } /* Comment.Preproc */ -body:not([data-theme="light"]) .highlight .cpf { color: #ababab; font-style: italic } /* Comment.PreprocFile */ -body:not([data-theme="light"]) .highlight .c1 { color: #ababab; font-style: italic } /* Comment.Single */ -body:not([data-theme="light"]) .highlight .cs { color: #e50808; font-weight: bold; background-color: #520000 } /* Comment.Special */ -body:not([data-theme="light"]) .highlight .gd { color: #d22323 } /* Generic.Deleted */ -body:not([data-theme="light"]) .highlight .ge { color: #d0d0d0; font-style: italic } /* Generic.Emph */ -body:not([data-theme="light"]) .highlight .gr { color: #d22323 } /* Generic.Error */ -body:not([data-theme="light"]) .highlight .gh { color: #ffffff; font-weight: bold } /* Generic.Heading */ -body:not([data-theme="light"]) .highlight .gi { color: #589819 } /* Generic.Inserted */ -body:not([data-theme="light"]) .highlight .go { color: #cccccc } /* Generic.Output */ -body:not([data-theme="light"]) .highlight .gp { color: #aaaaaa } /* Generic.Prompt */ -body:not([data-theme="light"]) .highlight .gs { color: #d0d0d0; font-weight: bold } /* Generic.Strong */ -body:not([data-theme="light"]) .highlight .gu { color: #ffffff; text-decoration: underline } /* Generic.Subheading */ -body:not([data-theme="light"]) .highlight .gt { color: #d22323 } /* Generic.Traceback */ -body:not([data-theme="light"]) .highlight .kc { color: #6ebf26; font-weight: bold } /* Keyword.Constant */ -body:not([data-theme="light"]) .highlight .kd { color: #6ebf26; font-weight: bold } /* Keyword.Declaration */ -body:not([data-theme="light"]) .highlight .kn { color: #6ebf26; font-weight: bold } /* Keyword.Namespace */ -body:not([data-theme="light"]) .highlight .kp { color: #6ebf26 } /* Keyword.Pseudo */ -body:not([data-theme="light"]) .highlight .kr { color: #6ebf26; font-weight: bold } /* Keyword.Reserved */ -body:not([data-theme="light"]) .highlight .kt { color: #6ebf26; font-weight: bold } /* Keyword.Type */ -body:not([data-theme="light"]) .highlight .ld { color: #d0d0d0 } /* Literal.Date */ -body:not([data-theme="light"]) .highlight .m { color: #51b2fd } /* Literal.Number */ -body:not([data-theme="light"]) .highlight .s { color: #ed9d13 } /* Literal.String */ -body:not([data-theme="light"]) .highlight .na { color: #bbbbbb } /* Name.Attribute */ -body:not([data-theme="light"]) .highlight .nb { color: #2fbccd } /* Name.Builtin */ -body:not([data-theme="light"]) .highlight .nc { color: #71adff; text-decoration: underline } /* Name.Class */ -body:not([data-theme="light"]) .highlight .no { color: #40ffff } /* Name.Constant */ -body:not([data-theme="light"]) .highlight .nd { color: #ffa500 } /* Name.Decorator */ -body:not([data-theme="light"]) .highlight .ni { color: #d0d0d0 } /* Name.Entity */ -body:not([data-theme="light"]) .highlight .ne { color: #bbbbbb } /* Name.Exception */ -body:not([data-theme="light"]) .highlight .nf { color: #71adff } /* Name.Function */ -body:not([data-theme="light"]) .highlight .nl { color: #d0d0d0 } /* Name.Label */ -body:not([data-theme="light"]) .highlight .nn { color: #71adff; text-decoration: underline } /* Name.Namespace */ -body:not([data-theme="light"]) .highlight .nx { color: #d0d0d0 } /* Name.Other */ -body:not([data-theme="light"]) .highlight .py { color: #d0d0d0 } /* Name.Property */ -body:not([data-theme="light"]) .highlight .nt { color: #6ebf26; font-weight: bold } /* Name.Tag */ -body:not([data-theme="light"]) .highlight .nv { color: #40ffff } /* Name.Variable */ -body:not([data-theme="light"]) .highlight .ow { color: #6ebf26; font-weight: bold } /* Operator.Word */ -body:not([data-theme="light"]) .highlight .pm { color: #d0d0d0 } /* Punctuation.Marker */ -body:not([data-theme="light"]) .highlight .w { color: #666666 } /* Text.Whitespace */ -body:not([data-theme="light"]) .highlight .mb { color: #51b2fd } /* Literal.Number.Bin */ -body:not([data-theme="light"]) .highlight .mf { color: #51b2fd } /* Literal.Number.Float */ -body:not([data-theme="light"]) .highlight .mh { color: #51b2fd } /* Literal.Number.Hex */ -body:not([data-theme="light"]) .highlight .mi { color: #51b2fd } /* Literal.Number.Integer */ -body:not([data-theme="light"]) .highlight .mo { color: #51b2fd } /* Literal.Number.Oct */ -body:not([data-theme="light"]) .highlight .sa { color: #ed9d13 } /* Literal.String.Affix */ -body:not([data-theme="light"]) .highlight .sb { color: #ed9d13 } /* Literal.String.Backtick */ -body:not([data-theme="light"]) .highlight .sc { color: #ed9d13 } /* Literal.String.Char */ -body:not([data-theme="light"]) .highlight .dl { color: #ed9d13 } /* Literal.String.Delimiter */ -body:not([data-theme="light"]) .highlight .sd { color: #ed9d13 } /* Literal.String.Doc */ -body:not([data-theme="light"]) .highlight .s2 { color: #ed9d13 } /* Literal.String.Double */ -body:not([data-theme="light"]) .highlight .se { color: #ed9d13 } /* Literal.String.Escape */ -body:not([data-theme="light"]) .highlight .sh { color: #ed9d13 } /* Literal.String.Heredoc */ -body:not([data-theme="light"]) .highlight .si { color: #ed9d13 } /* Literal.String.Interpol */ -body:not([data-theme="light"]) .highlight .sx { color: #ffa500 } /* Literal.String.Other */ -body:not([data-theme="light"]) .highlight .sr { color: #ed9d13 } /* Literal.String.Regex */ -body:not([data-theme="light"]) .highlight .s1 { color: #ed9d13 } /* Literal.String.Single */ -body:not([data-theme="light"]) .highlight .ss { color: #ed9d13 } /* Literal.String.Symbol */ -body:not([data-theme="light"]) .highlight .bp { color: #2fbccd } /* Name.Builtin.Pseudo */ -body:not([data-theme="light"]) .highlight .fm { color: #71adff } /* Name.Function.Magic */ -body:not([data-theme="light"]) .highlight .vc { color: #40ffff } /* Name.Variable.Class */ -body:not([data-theme="light"]) .highlight .vg { color: #40ffff } /* Name.Variable.Global */ -body:not([data-theme="light"]) .highlight .vi { color: #40ffff } /* Name.Variable.Instance */ -body:not([data-theme="light"]) .highlight .vm { color: #40ffff } /* Name.Variable.Magic */ -body:not([data-theme="light"]) .highlight .il { color: #51b2fd } /* Literal.Number.Integer.Long */ +body:not([data-theme="light"]) .highlight td.linenos .normal { color: #6e7681; background-color: #0d1117; padding-left: 5px; padding-right: 5px; } +body:not([data-theme="light"]) .highlight span.linenos { color: #6e7681; background-color: #0d1117; padding-left: 5px; padding-right: 5px; } +body:not([data-theme="light"]) .highlight td.linenos .special { color: #e6edf3; background-color: #6e7681; padding-left: 5px; padding-right: 5px; } +body:not([data-theme="light"]) .highlight span.linenos.special { color: #e6edf3; background-color: #6e7681; padding-left: 5px; padding-right: 5px; } +body:not([data-theme="light"]) .highlight .hll { background-color: #6e7681 } +body:not([data-theme="light"]) .highlight { background: #0d1117; color: #e6edf3 } +body:not([data-theme="light"]) .highlight .c { color: #8b949e; font-style: italic } /* Comment */ +body:not([data-theme="light"]) .highlight .err { color: #f85149 } /* Error */ +body:not([data-theme="light"]) .highlight .esc { color: #e6edf3 } /* Escape */ +body:not([data-theme="light"]) .highlight .g { color: #e6edf3 } /* Generic */ +body:not([data-theme="light"]) .highlight .k { color: #ff7b72 } /* Keyword */ +body:not([data-theme="light"]) .highlight .l { color: #a5d6ff } /* Literal */ +body:not([data-theme="light"]) .highlight .n { color: #e6edf3 } /* Name */ +body:not([data-theme="light"]) .highlight .o { color: #ff7b72; font-weight: bold } /* Operator */ +body:not([data-theme="light"]) .highlight .x { color: #e6edf3 } /* Other */ +body:not([data-theme="light"]) .highlight .p { color: #e6edf3 } /* Punctuation */ +body:not([data-theme="light"]) .highlight .ch { color: #8b949e; font-style: italic } /* Comment.Hashbang */ +body:not([data-theme="light"]) .highlight .cm { color: #8b949e; font-style: italic } /* Comment.Multiline */ +body:not([data-theme="light"]) .highlight .cp { color: #8b949e; font-weight: bold; font-style: italic } /* Comment.Preproc */ +body:not([data-theme="light"]) .highlight .cpf { color: #8b949e; font-style: italic } /* Comment.PreprocFile */ +body:not([data-theme="light"]) .highlight .c1 { color: #8b949e; font-style: italic } /* Comment.Single */ +body:not([data-theme="light"]) .highlight .cs { color: #8b949e; font-weight: bold; font-style: italic } /* Comment.Special */ +body:not([data-theme="light"]) .highlight .gd { color: #ffa198; background-color: #490202 } /* Generic.Deleted */ +body:not([data-theme="light"]) .highlight .ge { color: #e6edf3; font-style: italic } /* Generic.Emph */ +body:not([data-theme="light"]) .highlight .gr { color: #ffa198 } /* Generic.Error */ +body:not([data-theme="light"]) .highlight .gh { color: #79c0ff; font-weight: bold } /* Generic.Heading */ +body:not([data-theme="light"]) .highlight .gi { color: #56d364; background-color: #0f5323 } /* Generic.Inserted */ +body:not([data-theme="light"]) .highlight .go { color: #8b949e } /* Generic.Output */ +body:not([data-theme="light"]) .highlight .gp { color: #8b949e } /* Generic.Prompt */ +body:not([data-theme="light"]) .highlight .gs { color: #e6edf3; font-weight: bold } /* Generic.Strong */ +body:not([data-theme="light"]) .highlight .gu { color: #79c0ff } /* Generic.Subheading */ +body:not([data-theme="light"]) .highlight .gt { color: #ff7b72 } /* Generic.Traceback */ +body:not([data-theme="light"]) .highlight .g-Underline { color: #e6edf3; text-decoration: underline } /* Generic.Underline */ +body:not([data-theme="light"]) .highlight .kc { color: #79c0ff } /* Keyword.Constant */ +body:not([data-theme="light"]) .highlight .kd { color: #ff7b72 } /* Keyword.Declaration */ +body:not([data-theme="light"]) .highlight .kn { color: #ff7b72 } /* Keyword.Namespace */ +body:not([data-theme="light"]) .highlight .kp { color: #79c0ff } /* Keyword.Pseudo */ +body:not([data-theme="light"]) .highlight .kr { color: #ff7b72 } /* Keyword.Reserved */ +body:not([data-theme="light"]) .highlight .kt { color: #ff7b72 } /* Keyword.Type */ +body:not([data-theme="light"]) .highlight .ld { color: #79c0ff } /* Literal.Date */ +body:not([data-theme="light"]) .highlight .m { color: #a5d6ff } /* Literal.Number */ +body:not([data-theme="light"]) .highlight .s { color: #a5d6ff } /* Literal.String */ +body:not([data-theme="light"]) .highlight .na { color: #e6edf3 } /* Name.Attribute */ +body:not([data-theme="light"]) .highlight .nb { color: #e6edf3 } /* Name.Builtin */ +body:not([data-theme="light"]) .highlight .nc { color: #f0883e; font-weight: bold } /* Name.Class */ +body:not([data-theme="light"]) .highlight .no { color: #79c0ff; font-weight: bold } /* Name.Constant */ +body:not([data-theme="light"]) .highlight .nd { color: #d2a8ff; font-weight: bold } /* Name.Decorator */ +body:not([data-theme="light"]) .highlight .ni { color: #ffa657 } /* Name.Entity */ +body:not([data-theme="light"]) .highlight .ne { color: #f0883e; font-weight: bold } /* Name.Exception */ +body:not([data-theme="light"]) .highlight .nf { color: #d2a8ff; font-weight: bold } /* Name.Function */ +body:not([data-theme="light"]) .highlight .nl { color: #79c0ff; font-weight: bold } /* Name.Label */ +body:not([data-theme="light"]) .highlight .nn { color: #ff7b72 } /* Name.Namespace */ +body:not([data-theme="light"]) .highlight .nx { color: #e6edf3 } /* Name.Other */ +body:not([data-theme="light"]) .highlight .py { color: #79c0ff } /* Name.Property */ +body:not([data-theme="light"]) .highlight .nt { color: #7ee787 } /* Name.Tag */ +body:not([data-theme="light"]) .highlight .nv { color: #79c0ff } /* Name.Variable */ +body:not([data-theme="light"]) .highlight .ow { color: #ff7b72; font-weight: bold } /* Operator.Word */ +body:not([data-theme="light"]) .highlight .pm { color: #e6edf3 } /* Punctuation.Marker */ +body:not([data-theme="light"]) .highlight .w { color: #6e7681 } /* Text.Whitespace */ +body:not([data-theme="light"]) .highlight .mb { color: #a5d6ff } /* Literal.Number.Bin */ +body:not([data-theme="light"]) .highlight .mf { color: #a5d6ff } /* Literal.Number.Float */ +body:not([data-theme="light"]) .highlight .mh { color: #a5d6ff } /* Literal.Number.Hex */ +body:not([data-theme="light"]) .highlight .mi { color: #a5d6ff } /* Literal.Number.Integer */ +body:not([data-theme="light"]) .highlight .mo { color: #a5d6ff } /* Literal.Number.Oct */ +body:not([data-theme="light"]) .highlight .sa { color: #79c0ff } /* Literal.String.Affix */ +body:not([data-theme="light"]) .highlight .sb { color: #a5d6ff } /* Literal.String.Backtick */ +body:not([data-theme="light"]) .highlight .sc { color: #a5d6ff } /* Literal.String.Char */ +body:not([data-theme="light"]) .highlight .dl { color: #79c0ff } /* Literal.String.Delimiter */ +body:not([data-theme="light"]) .highlight .sd { color: #a5d6ff } /* Literal.String.Doc */ +body:not([data-theme="light"]) .highlight .s2 { color: #a5d6ff } /* Literal.String.Double */ +body:not([data-theme="light"]) .highlight .se { color: #79c0ff } /* Literal.String.Escape */ +body:not([data-theme="light"]) .highlight .sh { color: #79c0ff } /* Literal.String.Heredoc */ +body:not([data-theme="light"]) .highlight .si { color: #a5d6ff } /* Literal.String.Interpol */ +body:not([data-theme="light"]) .highlight .sx { color: #a5d6ff } /* Literal.String.Other */ +body:not([data-theme="light"]) .highlight .sr { color: #79c0ff } /* Literal.String.Regex */ +body:not([data-theme="light"]) .highlight .s1 { color: #a5d6ff } /* Literal.String.Single */ +body:not([data-theme="light"]) .highlight .ss { color: #a5d6ff } /* Literal.String.Symbol */ +body:not([data-theme="light"]) .highlight .bp { color: #e6edf3 } /* Name.Builtin.Pseudo */ +body:not([data-theme="light"]) .highlight .fm { color: #d2a8ff; font-weight: bold } /* Name.Function.Magic */ +body:not([data-theme="light"]) .highlight .vc { color: #79c0ff } /* Name.Variable.Class */ +body:not([data-theme="light"]) .highlight .vg { color: #79c0ff } /* Name.Variable.Global */ +body:not([data-theme="light"]) .highlight .vi { color: #79c0ff } /* Name.Variable.Instance */ +body:not([data-theme="light"]) .highlight .vm { color: #79c0ff } /* Name.Variable.Magic */ +body:not([data-theme="light"]) .highlight .il { color: #a5d6ff } /* Literal.Number.Integer.Long */ } } \ No newline at end of file diff --git a/backwards_compatibility.html b/backwards_compatibility.html index 9ac187b..aa6ac78 100644 --- a/backwards_compatibility.html +++ b/backwards_compatibility.html @@ -24,14 +24,14 @@ } @media not print { body[data-theme="dark"] { - --color-code-background: #202020; - --color-code-foreground: #d0d0d0; + --color-code-background: #0d1117; + --color-code-foreground: #e6edf3; } @media (prefers-color-scheme: dark) { body:not([data-theme="light"]) { - --color-code-background: #202020; - --color-code-foreground: #d0d0d0; + --color-code-background: #0d1117; + --color-code-foreground: #e6edf3; } } @@ -180,13 +180,15 @@
  • RDF and Friends
  • SOLID
  • Linked Data Fragments
    • +
  • HDT
  • Linked Data Platform
  • NanoPubs
  • Data Structures
    • @@ -212,7 +214,10 @@

    Ecosystem

    Drafting

    diff --git a/adapter/index.html b/codecs/hdf5.html similarity index 88% rename from adapter/index.html rename to codecs/hdf5.html index d482107..9e3d4b3 100644 --- a/adapter/index.html +++ b/codecs/hdf5.html @@ -3,11 +3,11 @@ - - + + - Adapter - p2p-ld 0.1.0 documentation + HDF5 - p2p-ld 0.1.0 documentation @@ -24,14 +24,14 @@ } @media not print { body[data-theme="dark"] { - --color-code-background: #202020; - --color-code-foreground: #d0d0d0; + --color-code-background: #0d1117; + --color-code-foreground: #e6edf3; } @media (prefers-color-scheme: dark) { body:not([data-theme="light"]) { - --color-code-background: #202020; - --color-code-foreground: #d0d0d0; + --color-code-background: #0d1117; + --color-code-foreground: #e6edf3; } } @@ -180,13 +180,15 @@
  • RDF and Friends
  • SOLID
  • Linked Data Fragments
    • +
  • HDT
  • Linked Data Platform
  • NanoPubs
  • Data Structures
    • @@ -212,7 +214,10 @@

    Ecosystem

    Drafting

    @@ -258,21 +263,22 @@
    -
    -

    Adapter#

    -

    Interfaces to other protocols and formats

    +
    +

    HDF5#

    +

    We are starting with hdf5 because our initial test case is the NWB format for neurophysiology data. This is a challenging initial test case because the data is heterogeneous, large, and the specification is written in an idiosyncratic format specification.

    +

    HDF has three primary types of objects:

      -

    • Files

      -
        -

      • hdf5

        • -

      • json

        • -

      • csv

        • -

      • mat

        • +

      • Groups

        • +

      • Datasets - contain the raw values in the file

        • +

      • Attributes - metadata about groups or datasets.

      -
      • -

    • HTTP

      • -

    • S3

      • +

      Datasets have additional properties:

      +
        +

      • Datatypes: binary representation of the data

        • +

      • Dataspaces: Layout of individual data elements

        • +

      • Properties: Additional information about the representation of the dataset, eg. chunked or contiguous

      +

      These map naturally onto triplets, where each group or dataset is a subject, attributes are properties and objects.

    @@ -289,14 +295,14 @@ - +
    Previous
    -
    Triplets
    +
    Codecs
     diff --git a/codecs/index.html b/codecs/index.html new file mode 100644 index 0000000..0bc1722 --- /dev/null +++ b/codecs/index.html @@ -0,0 +1,347 @@ + + + + + + + + + + Codecs - p2p-ld 0.1.0 documentation + + + + + + + + + + + + + + + + + Contents + + + + + + Menu + + + + + + + + Expand + + + + + + Light mode + + + + + + + + + + + + + + Dark mode + + + + + + + Auto light/dark mode + + + + + + + + + + + + + + + + + + + +
    +
    +
    + +
    +
    +
    p2p-ld 0.1.0 documentation
     +
    +
    +
    + +
    + +
    +
    + +
    +
    +
    + + + + + Back to top + +
    + +
    + +
    + +
    +
    +
    +

    Codecs#

    +

    Interfaces to file formats

    +

    We want to support three kinds of interaction with files:

    +
      +

    • References - treat files like abstract binary with some metadata indicating file type and a hash tree for the file

      • +

    • Introspection - Export some metadata from the file that indicates components of the file along with byte ranges. We want to be able to know what is inside the file without downloading it, but we keep the file separate as an out of protocol entity.

      • +

    • Ingestion - Export the metadata and the data contained within the file to triples. We also store some translation between the original binary file and the resulting triple through translation schema that allows us to update our triples if the files change and otherwise keep a strong link to the source, but otherwise enable forking/querying/etc. as if the data does not have an underlying file.

      • +
    +

    This is a challenging design balance, where we don’t want clients to need to implement a large number of codecs for different files - so they can fall back to the reference strategy as needed - but we also want people to be able to interact and import their files without needing to abandon longstanding practices or other infrastructure they might already have for using/creating them.

    +
    + +
    +
      +

    • Files

      +
        +

      • json

        • +

      • csv

        • +

      • mat

        • +
      +
      • +
    +
    + +
    +
    + +
    + +
    +
    + + + + + + \ No newline at end of file diff --git a/comparison/data/datalad.html b/comparison/data/datalad.html new file mode 100644 index 0000000..c461a1e --- /dev/null +++ b/comparison/data/datalad.html @@ -0,0 +1,357 @@ + + + + + + + + + + DataLad - p2p-ld 0.1.0 documentation + + + + + + + + + + + + + + + + + Contents + + + + + + Menu + + + + + + + + Expand + + + + + + Light mode + + + + + + + + + + + + + + Dark mode + + + + + + + Auto light/dark mode + + + + + + + + + + + + + + + + + + + +
    +
    +
    + +
    +
    +
    p2p-ld 0.1.0 documentation
     +
    +
    +
    + +
    + +
    +
    + +
    +
    +
    + + + + + Back to top + +
    + +
    + +
    + +
    +
    +
    +

    DataLad#

    +

    DataLad is a tool for managing datasets! It is built on top of git and git-annex for storage, and is capable of integrating with external hosting providers (through git annex)

    +

    It has a number of interesting extensions that we can learn from

    +
      +

    • crawler allows you to archive web pages

      • +

    • OSF Remote - example of an extension for interacting with OSF

      • +
    +
    +

    References#

    + +
    +
    + +
    +
    + +
    + +
    +
    + + + + + + \ No newline at end of file diff --git a/comparison/data/dmc.html b/comparison/data/dmc.html index bcf776a..2c204f3 100644 --- a/comparison/data/dmc.html +++ b/comparison/data/dmc.html @@ -3,7 +3,7 @@ - + @@ -24,14 +24,14 @@ } @media not print { body[data-theme="dark"] { - --color-code-background: #202020; - --color-code-foreground: #d0d0d0; + --color-code-background: #0d1117; + --color-code-foreground: #e6edf3; } @media (prefers-color-scheme: dark) { body:not([data-theme="light"]) { - --color-code-background: #202020; - --color-code-foreground: #d0d0d0; + --color-code-background: #0d1117; + --color-code-foreground: #e6edf3; } } @@ -180,13 +180,15 @@
  • RDF and Friends
  • SOLID
  • Linked Data Fragments
    • +
  • HDT
  • Linked Data Platform
  • NanoPubs
  • Data Structures
    • @@ -212,7 +214,10 @@

    Ecosystem

    Drafting

    @@ -269,23 +274,23 @@