How to add a new dataset

Add the table(s) on a server

If the dataset is not yet on the Internet, you need to put it on a web server.

Here we show an example for http://www.lexique.org maintainers:

• Connect to the server and go to the databases directory

    cd /var/www/databases
  

• Create a folder for the database, for example zzzz

    cd zzz
  

• Copy there the relevant files (csv, tcsv, xlsx, zip, …)

• To generate a .rds file (a binary format for faster loading in R), create a script make_rds.R like the following:

    #! /usr/bin/env Rscript
    require(readr)
    yyyyyyyyyy <- read_delim('xxxxxxxxxxxxxxxxxxxxxxx.tsv', delim='\t')
    saveRDS(yyyyyyyyyy, file='xxxxxxxxxxxxxxxxxxxxxxx.rds')
  

• Run the script to generate the rds file:

    chmod +x make_rds.R
    ./make_rds.R
  

• Create a link towards the rds file in the ../rds folder:

    cd ../rds
    ln -s ../zzzz/xxxxxxxxxxxxxxxx.rds .
  

• Restart the shiny server:

    sudo systemctl restart shiny-server.service
  

create a json description file

If you plan to use the data fetchers, it is necessary to create a .json file describing the dataset, and to push it to http://github.com/chrplr/openlexicon/datasets-info

Here is, for example, the .json file associated to Lexique3

{
    "_comment": "# Time-stamp: <2019-04-30 17:01:41 christophe@pallier.org>",
    "name": "lexique3",
    "description": "Lexique382 est une base de données lexicales du français qui fournit pour ~140000 mots du français: les représentations orthographiques et phonémiques, les lemmes associés, la syllabation, la catégorie grammaticale, le genre et le nombre, les fréquences dans un corpus de livres et dans un corpus de sous-titres de filems, etc.",
    "website": "http://www.lexique.org",
    "readme": "https://chrplr.github.io/openlexicon/datasets-info/Lexique382/README-Lexique.html",
    "urls": [{
            "url": "http://www.lexique.org/databases/Lexique382/Lexique382.tsv",
            "bytes": 25850842,
            "md5sum": "28d18d7ac1464d09e379f30995d9d605"
        },
        {
            "url": "http://www.lexique.org/databases/Lexique382/Lexique382.rds",
            "bytes": 5923674,
            "md5sum": "e3e5f47409b91fdb620edfdd960dd7a5"
        }
    ],
    "type": "tsv",
    "tags": ["french", "frequencies"]
}

Note: the filesizes (bytes) and md5sum are obtained on the command line by running

 ls -l *.{rds,tsv}
 md5sum *.{rds,tsv}

---

• Go to your fork of http://github.com/chrplr/openlexicon/ and add the json files describing the database, as well as some documentation — at a minimum a README.md file) describing the database – and issue a pull request.

• Optionaly, if you want to make the database accessible in openlexicon:

  • Modify the openlexicon apps that will use this database (issue a pull request).

  • Connect to the server and run:

    cd ~chrplr/shiny-server git pull

–

Back to OpenLexicon

Time-stamp: <2019-07-17 19:05:17 christophe@pallier.org>