data.world FAQs

Introduction

Our SQL documentation is a ground up guide to using SQL for querying datasets on data.world. Many of the sections have exercises at the end of them, and these sections and exercises make up our SQL tutorial. There are three sections in this tutorial:

Setting up the tutorial project

How to use the project

Content and exercises

Setting up the tutorial project

In order to do the exercises for our SQL tutorial you will need to have a data.world account. They are free, and you can get one here. If you already have an account on data.world, the previous link will take you to your home page.

The exercises in the tutorial are based on the dataset SQL tutorial exercise data.Once you are logged in to data.world you'll have access to the dataset, and you can create your own project to use with the exercises. You can do the exercises without creating your own project by following theTry querylink on each exercise, however we recommend saving your own project with answers for each of the exercises as some of the exercises are built on the results of previous exercises. Follow the steps below to create your project:

Go to the SQL tutorial exercise data dataset on data.world. Select the down arrow to the right of the Explore this dataset button and select + Create a new project

Exercise 32

Add your project name, leave all the other options as they are, and select Create project. On the next screen you can either enter a description or do that later.

After you have created your project, the first page you’ll see is the project workspace. If you aren’t familiar with it, please read our article on navigating the project workspace for an overview.

How to use the project

Follow the links in Content and exercises below to read each section in the tutorial. When you are ready to do the exercises, open the project you created for the tutorial and select+ Add -> SQL Query in the Project workspace:

You'll be taken to a new blank query window.If you need help on how to format your query or an introduction to data.world’s query tools, see the article Query basics.When you have finished writing and testing your query, select Save next to theRun query button. We recommend that you name your query with the number of the exercise:

If you accidentally write a query after clicking aTry querybutton you can save it to your tutorial project by selecting your project from the dropdown menu on the field Where will this query be saved?.

Content and exercises

The SELECT and FROM Clauses

Exercise 1

Exercise 2

Exercise 3

LIMIT

Exercise 4

WHERE and comparison operators

Exercise 5

Exercise 6

ORDER BY

Exercise 7

Exercise 8

AND, OR, and NOT

Exercise 9

Exercise 10

Exercise 11

Arithmetic Operations

Exercise 12

Exercise 13

IS NULL and IS NOT NULL

Exercise 14

LIKE and NOT LIKE

Exercise 15

Exercise 16

BETWEEN

Exercise 17

OFFSET

Exercise 18

Joins

Exercise 19

Exercise 20

Exercise 21

Exercise 22

CAST

Exercise 23

Aggregations

Exercise 24

Exercise 25

Exercise 26

GROUP BY and FILTER

Exercise 27

Exercise 28

HAVING

Exercise 29

CASE

Exercise 30

IN and NOT IN

Exercise 31

Working With Dates

Final Notes

Our SQL documentation and tutorial are works in progress and this list of exercises will continue to expand.

After uploading data to data.world, if your data contains a common, general type of information that data.world recognizes, we will suggest related information that you can use to enhance your data and research. As data.world processes your data, we will analyze each field's data type and format in an attempt to match it to any known data types within the data.world system. When a potential match is discovered, it will be indicated by an interactive green triangle displayed against that field:

this query

Clicking the corner will present a menu which indicates that potential matches exist for this column of data.

Match suggestions modal

Clicking on the highlighted menu item will presents a list of potential matches for this column. In many cases, this will only be a single item, although it is possible to have multiple matches as shown below.

Related columns

Adding a matched column will sometimes cause additional column choices to become available. In the above case, by selecting to include the "us_state" matched column, the user will be presented with a number of other "Related" columns. Related columns can be added to your table as well to provide additional levels of aggregation. Related columns generally represent containing entities. So in the case of States, Census Divisions and Regions contain the state:

Clicking "Add related column" will add this information to your table. Also note that columns can be renamed after adding them to your table using the pencil icon.

Updated table

Upon clicking "Done" the file will be reprocessed and new columns will be inserted. This could take a few moments depending on the size of the table, although it should generally be pretty quick.

The reprocessed table now contains the selected columns along with all of the original data columns. Note that if a new version of the original file is uploaded (with additional rows of data, or additional columns), the file will automatically be matched using the same algorithm.

Informational match popup

Now that we have our match columns, we can gain some additional context about these rows. Clicking on the blue "bubbles" will being up a small popup with some additional information.

While these pop-ups only provide some basic information today, we will be working to flesh these out over time. Our ultimate goal is for these to contain complete descriptions, maps, flags, population counts, etc.

Removing matched columns

Match columns may be removed at any time from the column header menu:

Query aggregation (advanced)

By including the containing information in this table, we now have additional data that we can aggregate across. Let's jump in and see how many states our data has in each of the various census districts. First, choose to "Query this file"

A simple aggregation sql query across this column tells us that we have two states in the Pacific Census Division.

Query join (advanced)

If you have two tables, both of which have been matched to the same class, then you can easily join those tables using SQL. In the following example, two tables which have been matched to states will join on the entity by default.

Match logic

Matches are discovered based on the content of the column, not the name of the column.

Example: If a column contains values such as 78703, 78731, 00501, and 24151, then it will be recognized as a Zip Code, even if the column is not named zipcode or postalcode.

Available matches:

You can find the full up-to-date list of matches in the ddw/ontology-v0 dataset. Specifically, gives a full accounting of available matches.

There is no restriction on file types that can be uploaded or downloaded on data.world, and a dataset can consist of any combination of files added to it. There are some size limitations, and files are handled differently based on the extension as follows:

Tabular files

Formats: csv, tsv, xls, xlsx

Tabular files are presented in a spreadsheet-style preview and we perform basic analyses on each of the columns:

size limits

The data is then queryable using SQL and SPARQL; take a look at Query basics for more info on getting started with querying.

To provide these querying capabilities and in line with our mission to connect the world’s data (by making it linkable), we’re converting it to RDF Triples, or graph data, under-the-hood. To learn more, check out our blog post on the matter and the W3C primer on RDF.

Excel files will include all of the underlying sheets in a tabbed interface. Only the tabular data will be included; other elements like pivot tables and charts will not be shown in the preview but they will still be available in the original file.

In addition to viewing a preview of the data in the table, you can also see the metadata for the table by clicking onSwitch to column overview. The ability to switch between the data preview and the column overview persists in the summary even after it's been saved. For more information about column overviews see our article Column overview.

Database file formats

Formats: sqlite

Database dumps will consist of multiple tables, and a schema that models the type information and the relationships between those tables. Each table will be represented as a data.world table, which can be previewed and queried naturally via our SQL engine.

RDF data

Formats: rdf, rdfs, owl, jsonl, nt, ttl, n3

These formats are serializations of RDF data - since RDF is the native data format for data.world’s platform, the statements in this file are simply loaded into the graph for the dataset or project that the file is added to. By uploading raw RDF data into a dataset or project, that data is searchable via the attached SPARQL endpoint. Take a look at Query basics for more info on getting started with queries. We show a preview of the contents of the file, including summaries of the classes, properties, and namespaces used in the file.

Structured files

Formats: json, ND-JSON, other 'sufficiently tabular' json files

When a JSON file has a "sufficiently tabular" structure, we will attempt to produce a table of data that represents the contents of the file. Common logging formats that include JSON arrays of simple objects or newline-separated JSON objects will generally work great with this interpretation. If the structure of the file is too hierarchical or inconsistent in nature, the file will instead be treated in its raw form - you can view or download the file, but it’s not queryable through our query engine.

Archive and compressed formats

Formats: zip, tar, tbz2, tbz, bz2, tgz, gz, -gz, z, -z

Archives that contain multiple files can be extracted and the first 50 files are stored in the dataset. Each extracted file is then handled using the criteria established for its extension. Please note that archives are not extracted by default. To do so, a Contributor must click on the ‘Extract’ button on the right-hand side of the archive.

Individual files that are compressed (i.e. foo.csv.gz) are decompressed and then treated as though the uncompressed file had been added directly.

Images

Formats: jpg, jpeg, png, gif, svg

Images are displayed in-line.

Source files

Formats: ipynb (version 4 and higher), js, r, py, as, apl, bash, bas, bat, c, cpp, cs, css, d, dart, diff, go, ini, java, julia, kt, lua, matlab, nasm, ml, perl, php, ps1, rb, scala, sql, tcl, ts, vim, yaml, xml, asp, jade, tex, less, sass, scss, Dockerfile

Source files are presented with full syntax highlighting where appropriate.

Documents

Formats: txt, html, md, pdf

The above document formats are rendered during preview. Other document types can be uploaded but are not available for preview.

Note that iframe embeds are not rendered in html files. Instead, try adding it as an Insight in a Data Project.

---

All other file types can be uploaded and downloaded as long as they are within the supported .

The size of data files you can store on data.world is set by youraccount plan. To see your file limits, go to your profile >settings> > billing. More information on free and paid accounts can be found here. Here's what we currently support:

Dataset Limits:

A dataset ingested by data.world may have a maximum size of 1GB and up to 250 individual files. Datasets from live connections have no size limit, nor do metadata management datasets created by metadata crawling.

Individual File Upload Limits:

The maximum size for an individual file is 1GB. If you have a file that is larger than that, try compressing the file to get it under thelimit, but note that it would then only be available for download due to size constraints.

Inference & Preview Limits:

Non-tabular files that can be previewedonly display a file preview if less than 40k. Images will be displayed beyond that limit if possible.

For xls / xlsx, the file must be less than 100MB uncompressed for us to support query and data preview functionality.

For other supported data files, we will providedata preview and query capabilities up to 1GB.

For deeper details we have tables with specific size limit and timeout information. Please contact us if your application requires a greater number of files or a larger maximum file size.

Have data on another site that you'd like to import todata.worldfor easy sharing, collaboration, and querying? No problem! As long as you have a direct URL to one of our supported file types and permission to access it, data.world will be able to import it easily. This is a great solution for importing data from the web, data portals, cloud storage apps, GitHub, and API endpoints! Even better, if the files change at the source, data.world can automatically update it.

To add data from a URL click theAdd data button on the dataset overview page and select theSync from URL option:

automated sync

If the URL does not require authentication to access it, all you need to do to add it to your dataset is enter the shareable link (provided by the data source) into the source URL field and selectContinue:

For sources where you need permission to access the data, first paste the URL, then choose authentication and select the appropriate option in the dropdown menu (OAuth, Token, or Username and password aka Basic):

Headers and POST body are used to make API calls to sites that support REST API. See the support docs on those sites for required values. Though it is possible to include authentication information in headers, we strongly recommend using theAuthentication setting as collaborators on the dataset or project can see information in headers (including logins and passwords), but not Authentication values.

When you have finished entering the required information from your URL, clickContinue andyou'll be prompted to name the data file as it will be seen on data.world. Choose carefully, as this name cannot be edited later. Changing the name would require deleting the connection and creating a new one with the new name.

If data.world encounters an issue with the source URL, we will display an error requesting you to verify the link and the settings. Hovering over the ? next to the error message will show you the exact error returned:

ClickingEdit will bring up the same dialog you used to enter the initial parameters so you can make changes:

Note that sometimes the error returned will say404 Not Found instead of403 Forbiddenif it's an authentication failure even though the URL is correct. This result is a security feature of the API.

Once your files are added, configure them to update regularly through the settings.

When you ingest a tabular data file on data.world it is run through a series of inspections to validate both the structure and content of the data in the file. If issues are found, the file is flagged with a warning. Warnings are indicatedby either a yellow triangle or a red circle, depending on the severity. The warning flag can be found on the dataset overview page under the name of the file:

Data Inspectors

or on the About this file section asInspectionson thedataset or project workspace for the file:

The number of warnings is listed to the right of the flag. By far the most common, yellow triangles are there to alert you to potential problems with the data that might affect your ability to query it, or warn you that sensitive data (social security numbers, phone numbers, email addresses, etc.,) was detected.

Very occasionally you will get a red flag which indicates that there was an error on ingest and data from the original file was lost. Possible reasons for the loss of data include:

The original file was corrupt.

There was a data type mismatch between the data type identified for the column and the data stored in it.

Data that you choose to connect to a specified linked data class had values that didn't match the linked data.

For a list of all the inspection warnings and errors, see the article Data Inspectors.

Whether you get a yellow warning or a red error, you have the option to correct it or ignore it. If you get yellow warnings, click on the flag for the warning dialog box view the warning types and locations.The dialog groups the errors by type so you can review them one kind at a time. Each type of warning is labeled with what kind of issue it is, how many were found, and the location of each.Some flags are for issues you already know about and don't wish to fix. Those warnings you can simply dismiss:

Note: Once you have dismissed a set of warnings it will not show up in the file again even--if you delete and reimport the file or update it. The ONLY way to get a list of all the warnings back is to delete the file and ingest it again with a different name.

If you wish to correct the issues with files that were originally added to data.world by a direct add, you can:

download the file from data.world

make the corrections (the locations in the warnings will help you find them)

re-upload the file using the same name - by using the same name, you'll overwrite the file on data.world (as opposed to creating a new file, which would occur if you changed the name)

For files that are synchronized from external services (such as cloud storage services), you will need to:

update the file in the source system

either select theSync now button from the details window:or

from the workspace, choose theSync now button on the right sidebar:

Sometimes changes that you make to the data dictionary will cause error warnings in the data. In the example below, one of the columns in the file being ingested holds the ages of shark-attack victims. Some of the values in the column are "20's", "30 or 40", etc. If I wanted to to restrict the data being imported to only integers so I could use arithmetic functions on it, I could go in to the data dictionary for the file after import and set the column type from string to integer. Doing this would immediately cause a red flag on the inspections as some of the data would be left out on re-ingest due to a datatype mismatch:

For a list of all the inspection warnings and errors, see the article .

There are many great projects and datasets on data.world, and it's likely that at some point you are going to want to use data from them in your own work. There are two different ways to reuse data on data.world: linking, and downloading and re-uploading. Which option you choose depends on a few factors:

Is the source data in a project or a dataset?

How well does the source data meet your needs?

Is the data either streamed or regularly updated?

If the data is in a dataset (as opposed to in a project), is well-documented, concise, and clean, you may very well want to link to it. However if you need to make changes to it, you'll need to download it, edit it, and re-upload it.

Some reasons you might choose to link to the dataset include:

You don't need to make any additions to the source dataset (e.g., adding extra columns with data.world linked-data fact tables)

The source dataset is really clean so you don't need to go in and clean it up

The dataset is well-documented with a good dataset summary, references to the original source, and a complete data dictionary

The dataset is automatically updated from an external source

Some reasons you might choose to download and re-upload data include:

You want to add columns from data.world linked-data fact tables(e.g., US census region, currencies, ICD10 medical codes, etc.,)

You only want to use a subset of the files in a dataset and don't want the rest of the files adding unnecessary complexity to your dataset or project

The data files would benefit from cleaning for clarity (e.g., removing blank columns, removing columns containing a single value, changing file or column names, etc.,)

The data files only exist in a project and not in a dataset

The data dictionary and/or dataset summary are incomplete and you do not have write privileges to the dataset.

The table below summarizes the differences between linking and downloading a file and re-uploading a data file:

Linked vs Reimported Data

Linked

Reimported

Can add to a project

X

X

Can add to a dataset

X

Can extend data with data.world linked-data fact tables

X

Can edit data dictionary and dataset summary

X

Must recreate the dataset summary and the data dictionary for every file in the dataset

X

Do not have to use all of the files in a dataset

X

Can reuse data dictionary and other metadata

X

Automatically updated from original dataset

X

Must include all the files in a given dataset

X

Adding files to a dataset

Once you have created a dataset, there are a few different ways to add data to it. You can:

add files directly from a local drive or the cloud

link directly to a data source through a url

Connect your data via an integration (check out our super connectors if you don't see your datastore listed)

Adding data files from a local drive

The easiest way to add files from a local drive is to drag and drop them onto the add data box. Drag and drop allows you to add multiple files to a dataset at once:

Adding files from a URL

You can also add files by selecting the Add data button or clicking anywhere in the Add data box. The Add data button opens the add data window which lists all the ways you can add data to a dataset. Note that in addition to theUpload from computeroption you still have the option to drag and drop files into your dataset.Upload from computer is similar to drag and drop, but you can only select one file at a time for upload.Dragging and dropping is much more efficient for adding multiple files from a local drive.

Adding data files from the cloud

If you want to add data files from the cloud you will need to configure your cloud service account to allow access by data.world. To configure a service, go to https://data.world/integrations/categories/import or click on theAdd data button from the dataset overview and following the link to theintegration page:

Once you have configured a cloud drive, it remains connected for further use. There is no limit to the number of cloud drives you can have configured.

You can select and add multiple cloud-based files at a time, and files sourced from the cloud can also be set to sync regularly--ensuring you'll always have the most up-to-date version of the file in your dataset.

Sync options can be changed at any time from the overview page of the dataset:

If you manually update a file (delete and re-upload it) or if a file updates automatically from a sync, all the previous versions of the file are preserved in data.world and can be downloaded at any time. In this way your data is preserved for auditing, accountability and versioning. To access past versions of your data go to the Activity tab on the dataset and select Versions, then click on the three-dot menu to the right of the version you wish to recover:

If you'd like to know how to link data through a URL see the article .

Create a dataset to house data, metadata, and supporting documentation when the data could be used in many different analysis projects. Create a project to house all of the work that goes into an analysis project, and link in the datasets to support your work rather than duplicating (see the article on datasets vs. projects for more information).

When uploading tabular data, we recommend a CSV file format over an Excel format (xlsx) as we can support larger file sizes for querying.

Remove any headers, footers, or notes outside of a single row of column headers from the data file. Include any removed content in the dataset summary or upload as a separate notes file within the dataset. Keeping the data file basic (machine readable over human readable) ensures data.world will import and analyze the data with the best accuracy.

Tag and document your data so that others will better understand and use the data.

Use the data inspector to verify your data has imported correctly and for a view into the data’s quality.

Ensure your dataset is within the data.world size limitations. You can also upload a zip file, and then attempt to extract it after upload if presented that option in the application. If you need data.world to support a larger dataset size for queries, please contact us and we'll do our best to accommodate.

Files within a dataset are displayed alphabetically, so if the files in your dataset should be displayed in a particular order, name them accordingly (01_*.xls, 02_*.pdf, etc.) or use the summary to take othersthrough your data and analysis.

Search first to see if the same dataset has already been uploaded, and if so, consider collaborating or linking directly to that dataset rather than uploading a duplicate. There’s nothing wrong with uploading your own copy, but sharing through collaboration or direct linking will keep that data’s ‘story’ in one place.

A dataset is the basic repository for data files and associated metadata, documentation, scripts, and any other supporting assets that should be stored alongside the data. Datasets are where all data is stored and documented for later sharing and use in projects. While there are many functional similarities between datasets and projects, we recommend that you store your data in a dataset and work with it--combine, query, analyze, draw insights, etc.--in a project.

When you create a dataset it might be because you have a database or other tabular data that you want to analyze and share. But data from a database isn't the only kind of data you can put in a dataset. Any file type can be saved there. Check out our article on supported file types for Information about various file types and the ways they are handled.

There are several ways datasets can be created:

Manually - we'll walk through that here

Via our API - instructions available in our API docs

Through super connectors like Stitch, KNIME, Knots, and Singer - instructions can be found in our integration documentation under super connectors

With Sparklebot -for data portals or enterprise companies, contact data.world to find out more about our tools to automate creation and syncing of your data library. This can be full data and metadata mirroring, or simply a catalog of your data sources with metadata and sample data where you'd like.

Creating a dataset

While logged in to data.world click on+New in the upper right corner of your window to create a new dataset and you'll be prompted to choose either a dataset or a project:

Creating a project

Choose Create new datasetand you'll be prompted to name the dataset, and set the ownership and, accessibility. If you are in one or more organizations, by default the owner field will contain the name of one of the organizations you are in. You can also set the owner to be yourself or any of the other organizations you are in by selecting the dropdown on the owner filed:

Dataset owner and permissions

If thedataset is intended to be used in the organization, it should typically be created with the organization as the owner. In this way the dataset benefits from the organization's service tier, permissions can be easily set based on the members of the organization, and datasets remain available within the organization even as individuals and permissioning changes. Permissions on a dataset owned by an organization can either be set to No one or to everyone in the organization:

If you are not in any data.world organizations, you will automatically be set as the owner of the dataset, and you can choose to keep the dataset private or to share it with the data.world community:

The number of private datasets you are allowed is determined by your user license--you can create as many public datasets as you would like. More information on account types and pricing are found on our pricing page. There are several factors to consider when deciding whether to make your dataset public or private:

When you make a dataset public you allow others to use that dataset in their own projects and build from it. They can't change your dataset in any way or even save queries to it, but they can use and share it.

Data that is public on data.world can be downloaded from data.world and used externally. If your data is proprietary or sensitive, it shouldn't be shared.

Publicly shared datasets add to the amount of information that is available to everyone for analyzing, visualizing and learning from

More information on permissions can be found in the article Understanding permissions.

Whatever the permissions are set at for the dataset will also pass through to any projects that use the dataset. So if the dataset is shared with no one then only you will be able to use it in a project, and if the project in which you include it is open to everyone, no one else will be able to see that dataset. Permissions can always be edited at a later time. After you create your dataset you can document your objective for it, add data to it, or continue on to the overview.

Adding files to a dataset

The files stored in a dataset can include more than just clean data. They can contain raw data, the scripts used to clean the data, the clean data, images, documentation--i.e., any information that would be useful for analyzing and understanding the data.See What file types can I upload? for more information about how data.world handles different file formats.

The easiest way to add files to your dataset is to drag and drop them into Add data box, but there are many other mechanisms for adding data. For more detailed information, see the articles Adding data files, and Adding files from a URL.

The overview tab on the metadata page is your main page for the dataset after creation. From here you can both add data to it and document the data:

Documenting your data consists of writing descriptions of the files, tables and columns, creating a summary, and adding tags. A starting point for information on all of these activities can be found in our overview to documenting your data. Once you've created a dataset you can either create a project to work with the data or add it to an existing project. For more information on projects, see our article .

There are several ways to get your data or metadata into data.world and there is no one right choice. There are benefits to each method, and which you choose will depend on several factors:

What format is your data in?

Where is your data currently located?

What is the size of your data?

How often does it update?

What are you going to do with it?

Whichever method you choose, the place you'll put your data is in a dataset. A dataset is the basic repository for data files and associated metadata, documentation, scripts, and any other supporting assets that should be stored alongside the data. Datasets are where all data is stored and documented for later sharing and use in projects.

In this article we'll look at the different places your data might live and how best to get it into data.world. The data sources we'll examine are:

Databases or data warehouses

Local files

Cloud-based storage (in sources like Google Drive, Box, Dropbox or S3)

Excel spreadsheets

Data from real-time sources

Via a URL or RESTful API

Corporate network

Databases or Data Warehouses

If you work with a JDBC-compatible database or warehouse, you can use our API to directly sync your data into a dataset. Below is a view of the databases supported by our API, and a current list can be found on our database integrations page.

enterprise tier

If you wish to leave your data at rest in your existing database or data warehouse, whether it’s on-premises or in the cloud, our Enterprise Tier supports virtualized access to that data source.

Local files

The best way to get files from your computer onto data.world is to upload them directly into a dataset. Files uploaded from a computer cannot be automatically synced or updated, but you can manually push new versions up to your dataset, replacing the previous version, as needed. When a new version is uploaded, the older version is still available for auditability and versioning. More about uploading data from local files and versioning can be found in our article Adding data files.

Cloud-based storage

Documents that are stored in cloud-based storage services (e.g., in Google Drive, Box, Dropbox or Amazon S3) can be easily added to data.world with one of our integrations and set to sync so that they update automatically:

As with manual updating, versions of files that are automatically updated are also kept for reference. More information about adding cloud-based files can also be found in the article Adding data files.

Excel spreadsheets

For Excel spreadsheets, data.world has created a specific add-in that's available on the AppSource or from within Excel. The add-on allows you to work with your data in Excel while at the same time sharing it in a dataset with others who may not have or use Excel:

See our Excel integration page for more information. Of course if you so choose you can always either upload your Excel spreadsheet into a dataset like you would any other file type, or you could put in a cloud service like Google Drive, Box or DropBox and add it to the dataset there so it can automatically sync between the two. Versions of Excel files that are uploaded or synced are also kept for future reference.

Data from real-time sources via streaming

You might have data that updates in real-time that you would like to put on data.world. This data could be something like log files, test metrics or tracking data. The best way to integrate this data into a dataset is to use data.world's streaming API. Unlike the methods previously mentioned which pull data from the source on a regularly scheduled basis, data brought in through the streaming API can be pushed into a dataset based on a change to the original data. Because it's triggered by data events and not random time intervals, using the streaming API is the best way to manage real-time data. You can read more about streaming in our API Quickstart guide.

For those less comfortable with working directly with an API, data.world also integrates with several superconnectors like IFFTTT, KNOTS, Singer or Stitch. While easier to use, they are less flexible and versatile than our own streaming API. You can see a full list of our superconnector integrations on our superconnector integrations page.

Data via a URL or RESTful API

Another common source of data is from a URL or RESTful API available on the internet. If you have a Google Sheets doc, e.g., you can add it to a data.world dataset. As long as the data is on a site that's publicly accessible, you can sync it to data.world--even if it's on a password-protected site with data.world's option to add from a URL. Detailed instructions for adding and syncing data from a url can be found in the article Adding files from a URL. If you do not own the data from the web that you'd like to bring into data.world, you can find out more about licensing and data in the article Licensing and data you found.

If you have data that is behind an API that you'd like to put on data.world--e.g., data from Salesforce, Facebook Ads, Google Ads, etc.--the best way to get it into a dataset is to use one of the superconnectors shown above. More information about our sales and marketing app integrations can be found here.

On-premise data

In addition to data that is available to data.world via cloud sources or APIs, some data that you might want to make accessible on data.world might only be available on your corporate network or behind a firewall. For customers with a need to catalog data behind a firewall, we make our Virtual Data Connector available as an appliance that can be hosted at your site and communicates with data.world via a secure bridge protocol. This option is available to our customers. If you have this need, please contact our sales team at [email protected] and they will help you with your options.

SPARQL (pronounced "Sparkle") is a powerful query language to retrieve, modify and make the best use out of linked data. It is recognized as one of the key technologies of the semantic web due to its flexibility as well as ease of joining complex data structures and detecting intricate patterns in data. Its also the query language upon which the data.world platform is based.

If you're new to SPARQL, don't worry! See our Getting Started with SPARQL tutorial to help get you started with the basics.

Query editor shortcuts for both SQL and SPARQL are available on data.world. Below is a list of the supported commands for Mac and Windows:

command + option + L

(ctrl + alt + L on Windows)

Automatically reformat your query to make it more readable.

command +shift + enter

(ctrl + shift + enter on Windows)

Automatically reformat AND run your query.

command+ enter

(ctrl + enter on Windows)

Run your query.

command+ S

(ctrl + S on Windows)

Save your query

A query template creates a graphical interface that simplifies running a query with different parameters. This allows an end-user to easily change the values that a query useswithout needing to access or understand the underlying SQL or SPARQL code.

As an example, let's take a look at the Data Professional Salary Survey and examine the most popular databases from that survey in 2018. To do that, I could use the following SQL query:

our API docs

For those with a bit of SQL knowledge, modifying this query to get the results for the year 2017 is trivial. One would just need to change the year value on line 3 from 2018 to2017.

But when aquery involves JOINs, UNIONs, nested queries, and aliases, the number of lines in the query could go from 5 lines to 50 or more.

Finding the desired value buried in the query becomes far more challenging. What if we'd like someone with no knowledge of SQL to change the year and run the query without our assistance?

Wouldn't it be nice of they saw something like this instead?

We can do that by creating aquery template.

Creating a query template

Creating query templates is possible in both SQL and SPARQL queries.In both cases, we'll need to create a variable to turn our standard query into a parameterized query (though creating a such a variable requires a different process in each).

Then it's just one more click to create that graphical query template form shown above.

To follow along, you can create a new project and then link it to the dataset noted above.

SQL

Declaringa variable

From within the project workspace, create a new SQL query

Click theNew template link on the top right of the query editor

This brings up a new window that shows example code for adding a variable

Click on theAdd a sample statement button on the bottom left to insert this into your query

Change the example values as needed

Referring to the example above, modify the example statement to read:

In this case, ?year is the variable name, INTEGER is the data type, and 2018 is the optional default value.

If no default value is specified and the query is run without entering a value manually, the query will return an error.

Allowed data types are: STRING, INTEGER, DECIMAL, BOOLEAN, DATE, TIME, and DATETIME

Additional DECLARE statements can be added to create more parameters in the query.

Using the variable in the query

Now that the variable exists, choose where the variable will be used in the query - this will often be in the WHERE clause of the query - though it could also appear in an ORDER BY, LIMIT, or other clause . In our example, we'll use it in the WHERE clause on line 5:

Note: This example is a bit complicated because the survey_yearhas the data type YEAR in our database, but we can't declare a variable with that type. Instead, we declare our variable as a INTEGER and then used the CAST function to convert the value of survey_year from YEAR to INTEGER to make the comparison on Line 5.

Adding Comments

Comments added before the DECLARE statement will display in the final query template form. The creator of the query can provide instructions and present valid input values.

Denote comments by adding the # or -- characters before the desired text:

Creating the query template

After declaring and referencing the variable(s), click the "Preview query template" link that now shows abovethe query editor:

This will create the graphical form where an end-user can enter the desired valuefor the variable. We'll now see that friendly form originally shown in the Overview section:

Creating this query template will lock the query from further edits. To unlock it to make a change, just click the Edit query link above. If another user owns the query, then you will need to save a copy of it after making any edits.

That completes SQL query template - but keep in mind that you can create additional variables to make a more powerful, flexible query.

SPARQL

The SPARQL syntax can be a bit more difficult to understand for those who are unfamiliar. For a great primer, check out our SPARQL tutorial !

To get started on our SPARQL query template, create a new SPARQL within your project. We can mimic the SQL query above with the following SPARQL query:

Declaring a variable

Due to the syntax of SPARQL, a query will contain at least one variable by default.

To turn that into a parameter that can be used in a query template, change the character in front of the variable name from ? to $. Make sure to perform this change for every occurrence of the variable within the query.

A new variable can also be declared by starting its name with with$ character.

In our query, we'll do the latter by creating a new variable $year_of_interest andreplacing the value2018in the FILTER functionon line 9.

Setting default values

In that form we just created, you may have noticed we did not see a default value populate.

Setting an optional default value for a variable is more complicated in SPARQL than SQL. To do that, we need to add a conditional BIND statement on the first line of the WHERE clause. The general format of that statement is:

BIND(IF(BOUND($variable), $variable, <DEFAULT_VALUE>) AS $variable)

In our case, let's go back into the editor and add a new line on line 7:

BIND(IF(BOUND($year_of_interest), $year_of_interest, 2018) AS $year_of_interest)

Although this will set the default value, that value will not be shown in the textbox in the query template form as it does in a SQL - it will be hidden from the end-user. Even though that value is not visible, the query will now use 2018 as the default year unless otherwise specified.

Adding comments

Comments can be added to a SPARQL query template as well. Only the # character (and not the -- characters that are available in SQL) can be used to specify a comment. The query template will show all comments before the first SPARQL statement (typically a PREFIX statement).

Since the query template won't show any default values, making a note of such values in the comments is a great idea.

Creating the Query Template

Clicking thePreview query template link above once more will generate our final template:

Just like the SQL version, this query will be locked from edits until the "Edit Query" link above is clicked.

Sharing

Queries that are executed with a new parameter value may be shared with others by simply copying the URL and sending it to another user. If parameters have been filled in, then these parameters will be encoded into the URL for sharing.

API

Parameterized queries can also be made through the data.world public API. Note that non-parameterized queries can be executed with a GET request to the same endpoint, but in order to specify the parameters, a POST request must be made with the parameters specified in the body of the request.

Example curl command:

curl -X POST \

https://api.data.world/v0/queries/${DW_QUERY_ID}/results \

-H 'Accept: application/json' \

-H 'Authorization: Bearer ${DW_API_TOKEN}' \

-H 'Content-Type: application/json' \

-d '{

"parameters" : {

"parameterName1" : "parameter value 1",

"parameterName2" : "parameter value 2" }

}'

See for more information.

Introduction

A query, in the context of data, is a request for information written in a query language. A query language uses a specified set of terms in a specific order with proscribed punctuation. The query languages used on data.world are SQL and SPARQL. While data.world is built on semantic web technologies that are best queried using SPARQL, we recognize that most people are more familiar with SQL. For that reason, we've developed our own version of SQL to easily query data.world datasets. If you are new to SQL, we have documentation that willwalk you through everything from 'what is a database' to 'how do I join all these different tables together so I can query them all at the same time'. There is also a tutorial based on the exercises in the documentation. For those who would rather work with SPARQL, data.world also has a tutorial for SPARQL that will get you up to speed regardless of your skill level.

In this article we'll go over how to use the query panel in the project workspace for SQL queries including:

Creating a new query

Auto-complete of terms and objects

Click-to-copy for columns and tables

Pop-up help text

How to auto-format queries

Troubleshooting

Saving queries

Saving queries to datasets

See the article Quickstart to navigating the project workspace if you would like an overview of the entire project workspace.

Creating a new query

To write a SQL query against data in a project go to the upper left corner of the workspace and selectSQL Query from the+ Add menu:

Working with queries

When the query editor comes up in the center panel of the screen, the Project Schema also comes up in the right side panel. If the project schema isn't showing, select the arrow to the right of theRun query button to expand the sidebar:

Auto-complete of terms and objects

As you type your query, the editor suggests SQL terms--operators, aggregations, and functions--for you. Hitting 'enter' will autocomplete the highlighted suggestion in the list. Arrow down or up to choose another option:

Click-to-copy for columns and tables

You have the option of typing in the table and column names for the data you are querying, but you can also use the click-to-copy feature on the project schema to copy the names for you so you don't have to worry about typos.

Pop-up help text

Help text is provided for all SQL functions and aggregations when you hover over the function name:

How to auto-format queries

While you can type on one line type your SQL query all on one line--it is perfectly legal and will run--it's good practice to split it up based on clauses and punctuation. The line breaks make it easier to see where you may have forgotten a comma or some other element of the query, and it also allows us to give you better error messages if there are problems with your query.

Here's an example of a query before and after running it through our auto-formatter:

To auto-format your query, use the keyboard shortcut Cmd + Option + L. If you forget the command there is a dropdown menuto the right of theRun query button which has a link to a list of all the keyboard shortcuts.

From the dropdown menu you can also manage your query (rename, copy, etc.) as well as get to our SQL tutorial for additional help.

Troubleshooting

If you try to auto-format your query and it doesn't work, that's a sign that you have an error in the query. If you run it you'll get an error message telling you what the problem is and where to find it:

In the above example the query is written to search for all the records that have the value "won" in the deal_stage column in the sales_pipeline table. However the way the query is written it looks like it is searching for records where the value in the deal_stage column matches the value in another column named Won. For "won" to be recognized as a value, it has to be surrounded by double-quotes. The error message returned from running the query tells you that the problem is in the 4th line and the 36th character (column). The 36th character should have been the opening " on the 4th line:

Saving queries

When your query is complete, save it for later use by selectingSave to the left of theRun query button. You'll be prompted to name the query, add a description, determine where to save it, and indicate who can see it:

:

It's good practice to give the query a short descriptive name for both your own later use and for use by others working on your project. Adding a description also makes your query more useable as the description shows up on the resource card returned from using search on data.world.

Saving queries to datasets

In some cases you might want to save your query in its dataset rather than in a project. Most of the time you'll want to work with your data--including querying it--from within a project. However if you do want a query to be available in every project that uses a certain dataset you can save the query to the dataset. When you select Save, instead of selecting the default+ New project, if you click the menu icon to the right you'll see that you can save the query directly to the dataset:

For information on all the ways you can use the results of a query see the article .

Queries against your data are the building blocks for an analysis of the data. We covered the basics of writing your own queries in the article on query basics. In this article we'll look at all the things you can do with an existing query including how to:

Use other people's queries

Copy and save to dataset or project

Make a template from the query

Download query results

Save results to a dataset or project

Work with third-party applications

Use other people's queries

As you work in projects with other people, you might find a query that you want to use and edit, but it's not your query so you don't have permissions to save changes to it. The easiest thing to do is to make a copy of the query by selecting the three dots to the right of the query name in the project workspace and selecting Save a copy:

Integrations page You'll be prompted to name the query and write a description. For tracking lineage it's a good idea to include the name of the original query in the description. Other save options include where--the current project or the original dataset--and who can see the new query (hidden by the location selection in the image below:

Copy and save to dataset or project

Normally the choice would be to save the new query in the current project because then it would show up in the list of all the queries and would be limited to contributors on the project. However if the query would be useful to others who create projects from that dataset, it might make more sense to save it directly to the dataset. Saving it to the dataset enables anyone with access to the dataset to use it (unless you also set it to share with no one). It's also the easiest way to save the query to another project that is linked to the same dataset. One note: queries saved to datasets show up in the Connected datasets section of the Project directory in the left sidebar--not under Queries: Alternatively you could copy and paste the query into a new query window in the other project and save it directly there.

Make a template from the query

Another way you might want to modify a query is to make a template from it so you can change the input strings to various parts of it. For example we could turn the query we just saved as Animal breed query into a query template by selecting theNew template link at the top right of the query panel:

A dialog window will pop up giving you basic instructions to how query templates work and linking you to the documentation for query templates:

SelectAdd a sample statementa sample of the DECLARE statement used in query templates, a sample comment, and a link to the documentation will be inserted in the top of your query window:

With just a couple of modifications to the SQL you can have a query where other users can input values for animal typeswithout ever having to know SQL themselves. Here we moved the comment to the top line so it would show up as a prompt in the input area, and we indicated where in the original query the input value would go (in the WHERE clause):

You can preview the query to see what other users will see and to test the query yourself by selecting thePreview query template link in the upper right corner of the query panel. Then fill in a value and hit Run query:

For more detailed information on query templates see the article on using query templates.

Download the query results

Another way to use a query is to download the results from it in either a csv or an xlsx file. After running your query, click theDownloadlink below the query window and select the option you want:

Save results to a dataset or project

Another option for working with your query results is to save them as a tabular file in your current project, in an existing dataset, or in a new dataset:

There are many reasons you might want to save the results of your query in a new tabular file. See the article on saving query data to a new file for more information.

Two other ways you can use the results of a query are to make them available for download from a URL, or to embed them on a Markdown page. Both options are found in theDownload menu underCopy URL or embed code. The URL is a download link for a csv file of the results and the embed code option renders the results table in a Markdown page.

work with third-party apps

There are many applications you can use to analyze and display the results of your queries, and we make it easy to work with your favorites. You can find out about the applications we integrate with on our integrations page. If you don't have any integrations configured yet you can still make visualizations of your results with our built-in Chart Builder app:

For instructions on how to use Chart Builder see one of the following articles on it: an introduction to Chart Builder, using the Vega-Lite editor, and incorporating multiple columns of data in your Chart Builder charts.

If you already have an integration configured that you want to use or want to integrate a new application, the name of the first application in your list of integrations will show up as a menu to the right of theDownload button:

You can find out what integrations we currently offer and how to use them on our .

Have a dataset or project you'd like to share with a colleague for review or contributions? Hunting for some missing data to complete your projectand want to put out the request for someone in the data.world community to help?

From a dataset overview page, go to theContributors tab: Understanding permissions

Or if you're on a project overview page, go to thePeople tab:

From those tabs you can click the Invite button on the right tosend a request to any data.world user or organization to become a contributor on your dataset or project.To add a contributor, simply start typing their name or data.world username and you will see a list of matching users for you to pick from. Want to add someone who is not on data.world yet? No problem, just enter their email and we'll send them an invite. Select the desired permission level (read more at), then click the Invite button to send out the request.

After a contributor has been added, you can also return to People orContributors tab to control their level of accessor remove access entirely.

Owners of datasets and projects can invite specific users to contribute, or approve incoming requests from users who want to contribute. Either way, the owner controls what each contributor can do by granting three levels of permissions:

View only

View + edit

View + edit + manage

See our article on changing permissions for instructions on how to allow others to access to your datasets and projects. Here's what each permission level will allow a contributor to do:

View only: primarily used for private datasets and projects, this allows the user to simply view the dataset or project. As part of that, the contributor can:

Download any of the files.

Query the data and export results.

View and comment in either public or private discussions.

Create new discussion topics.

View + edit: in addition to the view-only permissions, the contributor can:

Make edits to descriptions and summaries.

Add and remove tags.

Add and remove files.

Replace files by uploading new versions with the same name.

Modify file and column descriptions.

Modify license type.

Switch the dataset or project between open and private.

Publish queries for others to use.

View + edit + manage: The contributor will have full admin controls to the dataset or project. In addition to the view + edit permissions, they can:

Delete the dataset or project.

Add, remove and modify contributors.

Suppose you are part of an organization on data.world (see Creating an organization for more info) and would like to create a new project or dataset. You can choose to create the asset in your own account or within the organization's account.

In both cases, you will maintain control over the dataset or project. You can invite contributors, change visibility from private to public, and edit it to your heart's content.

Organization owned

When you create a dataset or project with the organization as the owner, you can choose whether to share it with the entire public, just the organization, or with no one at all. When you choose to share it with the organization as in the following image, the organizations members will receiveView onlypermission by default (read more about permissions at Understanding permissions ).

To change the permission levels later, go to thePeople tabfrom the project overview page or theContributors tab from a dataset overview page. Find the organization and click on the dropdown menu underAccess to modify the permissions.

If you had chosen to make the project completely private when you had created it within the organization, then you will need to go to that samePeople orContributors tab to share it later. In that case, you'll see that the organization has a convenientInvite all members button next to their name. The organization will be added automatically - there is no need for an admin to accept the invitation. Again, those members will be given View access by default, but you can change that the same way as the last example.

Member owned

Individuals can share a dataset with an organization exactly the same way they share with other individuals. Just go to the Contributors tab on a dataset or People tab on a project, and use the Invite button toadd the organization.

Once added, the organization's administrators will receive a notification they can accept or reject. If they accept the invitation, the dataset will then be shared across the organization's top-level members using the same permissions the organization received.

When a dataset is set to be discoverable, it is listed in public search results and its metadata (description, summary, creator, and license), contributors, and discussion can be viewed by anyone on data.world. None of the individual files or tables in the dataset are visible, however.Only files that are explicitly marked as ‘Preview’ files are viewable by end-users until the end-user is granted read or write access to the dataset. The reason for the discoverable flag is to expose the existence of a dataset to others who might have a use for it, while at the same time maintaining control over who has can access the data in it. It is a useful tool for making users in other groups aware of the dataset so they can be granted permissions to it on an individual or group basis.

Setting a dataset as discoverable

Only a dataset that is owned by an organization (as opposed to an individual user) and which has its permissions set to private can be made discoverable. When a dataset is owned by an organization but set to private, it is not automatically shared with anyone else in the organization except the admins, even though it is set up for easy sharing with the entire organization at a later time from the Contributors tab. Making a dataset discoverable is done after the dataset has been created from the Settings tab:

reating a sample file for preview on a discoverable dataset

When a dataset is discoverable this is what it looks like to everyone who doesn't have explicit permissions to it. Notice the indication under the summary that there are files in the dataset to which the viewer doesn't have permission:

When someone wants to see the rest of the dataset, they select the Request access button in the upper right of any tab on the dataset. The creator receives notice of the request in email and can then either approve or disapprove it:

Adding previews of data to a discoverable dataset

By default none of the files in a discoverable dataset can be previewed on the overview tab. However If you would like a sample of the data to show on the discoverable dataset, you can create sample files that are then marked as previewable. See our article on c for details.

With discoverable datasets we introduced the ability to make a the existence of a dataset available to the general data.world community without exposing any of the data in it. In this article we'll discuss how to take this feature and extend it even further by adding sample data files that can also be viewed. Users can view the samples to determine if they want to request access to the full dataset. A preview of the sample file is visible on the dataset overview page. If the user evaluating the data would like to see more than the preview, the file can be downloaded and viewed.

Considerations

There are different ways to create sample preview files. A sample may have all of the columns as the original file, but not all the rows. Or it may have only some of the rows and some of the columns--columns with sensitive data having been removed.

Creating a sample file with all the columns

Starting from the overview page of your discoverable dataset, the easiest way to create a sample with all of the columns and only some of the rows is to select the Explore dataset button on the top right of the dataset overview page. Then select the file you would like to preview from the list in the left sidebar and click Query:

All you need to add to the sample query you are presented is a line at the end with a LIMIT clause in it setting the limit of the number of rows returned by the query to five (or however many you would like to be available. Note that only five will be previewed but the rest are available for download):

Hit the Run query button, thenDownload andSave to dataset or project:

It's a good idea to save the resulting table so that it's easily identified as a sample of a data file, not the file itself. The current dataset name will automatically be populated in the Dataset/Project field:

After you have added the file to your dataset you can go back to the dataset overview page by selecting View dataset from the left sidebar menu:

Then scroll down to the sample file and select the three dots on the right to edit its metadata:

Check thePreview option and save:

Creating a sample file with some columnsremoved

The process for creating this sample file is exactly the same as for the previous file except for one exception: Instead of using the * wildcard in your SQL query for "all columns", you need to list out the columns you want to include and then finish with the LIMIT clause as before. Here is an example:

SELECT sales_per_year.base_product,

sales_per_year.yearly_sales,

sales_per_year.statusid

FROM sales_per_year

LIMIT 5

each file can also be flagged as discoverable from the file metadata, accessed from the three dots to the right of the file name on the Overview tab:

Check the Preview box to make a random 5-line sample of the file visible from the overview page of the dataset:

Files that have been made previewable are flagged with the Preview label on the overview tab in the creator's view:

Note: Even though the file cannot be accessed in data.world until permission has been granted by the creator, it can still be downloaded by anyone. For this reason it's best to make a sample file derived from the original data and flag it as discoverable rather than make the original file previewable:

Chart Builder is a quick and easy tool for creating visualizations of data on the fly, but there is one thing that isn't easy to do with it: include data from more than one column in your graph. This limitation can be a real problem if, say,you want to look at both the high and low temperatures on the days when Bigfoot was sighted. Or if you want to have a graph with gender, attack type, and fatality in shark attacks so you can see if there is any correlation between them.Though you can easily run the queries to display the data, you can't obviously render it all at the same time in Chart Builder. However, though it is a bit tricky and requires the use of UNPIVOT, you can build visualizations in Chart Builder that include data from more than one column in a query.

In this article we'll use the Project Monsters Among Us to show how to include two related fields in a visualization, and Analysis of shark attacks by region and species to do a little fancier combination of multiple columns of unrelated data into one visualization.

How to show data from two related fields in a Chart Builder visualization

There is a query called High and low temperatures on the dates of Bigfoot sightings in the Monsters project that returns a simple table with three columns:

SQL documentation

Click on the Chart icon above the results to build a quick visualization from the query results. Set the X axis to date, the Y axis to temperature_low, and you have a visualization, but where do you put temperature_high?:

Looking at the data, the solution is to put both the high temp and the low temp values in the same column and call it temperature, and to have another column called temp_value that would indicate whether the temp shown is a high temp or a low temp for the day. Fortunately, this kind of data reorganization where columns get collapsed into rows is what the SQL UNPIVOT command does. Here is the original query rewritten to use UNPIVOT to collapse the high and low temp columns into one column, and the resulting table:

SelectChart to use Chart Builder on the results of the query, set the marks to Circle,the X axis todate, the Y axis to temperature, the color totemp_type, resize the chart toto 640 X 700, and you'll have this visualization :

Combining multiple columns of unrelated data into one visualization

In this example we have a query in the projectAnalysis of shark attacks by region and speciesthat returns dates, type of attack, gender of the victim, and whether the attack was fatal or not:

To get a quick visualization of it selectChart, setCircle for marks,year for the X axis (you might have to open the options and set the type to Ordinal), COUNT (*) for the Y axis, andGender for the color. Once it's been resized you get this chart:

As in the last example, there's no way to include attack-type or fatality data. However, a redo of the original query with UNPIVOT combines all the data into one column ready for Chart Builder:

Note: Even though there is a warning that only the first 10,000 rows of the results are displayed, when we chart the query with chart Builder, all the data is used in the visualization.

The chart from the query is built the same as before. SetCircle for marks,year for the X axis (you might have to open the options and set the type to Ordinal), COUNT (*) for the Y axis. Set Type for thecolor. Once it's been resized you get this chart::

If you want to try unpivoting some queries on your own and charting them, there are a couple more-- Provocation and gender in shark attacks, and Provocation and fatality in shark attacks --saved on the project that you can use.

Additional information about UNPIVOT can be found in our .

Chart Builder uses Vega-Lite, which provides a JSON syntax for creating and styling visualizations. While the Visual Builder interface within Chart Builder on data.world allows one to quickly generate a simple chart, using the Vega Lite editor allows extensive customization of the appearance of the chart.

This article will discuss some basic style modifications that are not provided in the Visual Builder :

Styling the chart title

Styling the axis labels

Styling the axis titles

One important note - once you modify a chart using the Vega Lite editor, the Visual Builder will no longer be accessible. Customize the chart as much as possible in the Visual Builder first before switching to the Vega Lite editor for fine-tuning to keep yourself from needing to do extra work.

This article assumes that you have already enabled the Chart Builder integration on your data.world account. If you have not already, you can enable it from the integrations page while logged into your account.

For a primer on using Chart Builder, please see the Data visualization with Chart Builder.

Getting started

As an example, I've created a project based on a dataset from the US Department of Energy regarding types of energy production throughout the US.

Open up the following query saved to that project: Top 10 states by residential solar energy production

You can then open these query results in Chart Builder using the dropdown menu above the results pane: full documentation

Title the chart "Rooftop photovoltaic energy production by US state (top 10)" by clicking on the text that saysUntitled chart above.

To the left of the chart, configure the the X axis to use thestate field and the Y axis to use thegwh field.

Click on the Optionsdropdown for the X axis, and under theSort section, chooseDescending byy - gwh.

You will then see the same chart as below:

That's a good start, but it's a bit bland and would be difficult to read if projected onto a screen across a conference room. Let's get to work!

Styling the chart title

By default, a Chart Builder adds the title automatically, but does not provide any graphical way to style it. Click on the Vega Lite editoron the top left and you'll see the following entry near the top:

"title": "Rooftop photovoltaic energy production by US state (top 10)"

To transform the title into a field that we can customize, make it into a JSON object by adding curly braces and adding the text property:

"title": {

"text": "Rooftop photovoltaic energy production by US state (top 10)"

}

The title will look the same as before, but we've laid the foundation for further styling by turning it into a JSON object.

Alignment

Theanchor attribute determines the horizontal alignment of the title. Options include:

start

middle

end

In our example, let's align the title on the left side of the chart:

"title": {

"text": "Rooftop photovoltaic energy production by US state (top 10)",

"anchor": "start"

}

Font

We'll use the following attributes to style the font used for the title:

font - the name of the font

fontSize - the size of the font in pixels

color- color of the font, given in a CSS-compatible hex code or color name

This attributes are great for matching the chart to an organization's own branding guidelines. Make the following update to give the title that authentic data.world feel:

"title": {

"text": "Rooftop photovoltaic energy production by US state (top 10)",

"anchor": "start",

"font": "Lato",

"fontSize": 24,

"color": "#355D8A"

}

Offset

Our title is looking much better now, but there isn't much space between it and the chart beneath it. Give it some breathing room by adding theoffsetattribute. The offset value is the number of pixels between the title and the edge of the chart.

"title": {

"text": "Rooftop photovoltaic energy production by US state (top 10)",

"anchor": "start",

"font": "Lato",

"fontSize": 24,

"color": "#355D8A",

"offset": 40

}

And here's the result so far:

Now let's improve those other text fields...

Styling the axis labels

To modify the axis labels (in this case, thenames on the x-axis andnumbers on on the y-axis), we'll add some additional attributes within theconfigobject already present in our Vega Lite editor. By default, theconfig object will look like:

"config": {

"background": "#ffffff",

"padding": 20,

}

Within thatconfigobject, add a new object calledaxisto modify the labels on both the X and Y axes at the same time. That object will accept a number of attributes; we'll use the following:

labelFontSize - label font size in pixels

labelFont - label font name

labelColor - color of the label font, given in a CSS-compatible hex code or color name

Ourconfig object now looks like this:

"config": {

"background": "#ffffff",

"padding": 20,

"axis": {

"labelFontSize": 20,

"labelFont": "Lato",

"labelColor": "#6290C3"

}

}

We modified labels for both axes above, but we can also style axes singly as well. The state names under the X axis are difficult to read with their current orientation, so change that by adding an axisX object within theconfig object.

Use thelabelAngle attribute to control the angle of those labels, providing the number of degrees to rotate them.

"config": {

"background": "#ffffff",

"padding": 20,

"axis": {

"labelFontSize": 20,

"labelFont": "Lato",

"labelColor": "#6290C3",

"titleFontSize": 24,

"titleColor": "#333D49",

"titleFont": "Lato"

},

"axisX": {

"labelAngle": 40

}

}

Styling the axis titles

Our labels are much more readable now, but we need to update thosestate andgwh axis titles as well. For those, specify the following attributes within the config>axis object:

titleFont -axis title font name

titleFontSize -axis title font size in pixels

titleFontColor - color of the axis title font, given in a CSS-compatible hex code or color name

"config": {

"background": "#ffffff",

"padding": 20,

"axis": {

"labelFontSize": 20,

"labelFont": "Lato",

"labelColor": "#6290C3",

"titleFontSize": 24,

"titleColor": "#333D49",

"titleFont": "Lato"

}

}

Finally, let's edit the title text for each axis (e.g. state and gwh). This will use the column name from our query results by default. In our case, those titles aren't so bad, but in many cases they'll be difficult to read and full of underscores and abbreviations.

Navigate to theencoding object within the Vega Lite editor. It will have a number of nested objects below it already. Find the nested object encoding>x. Add and attribute called title and providethe desired name to that attribute - in this case, give it the value "State Name".

Also add atitle attribute under encoding>y and provide the value"Gigawatt hours (GWh)".

Here's our newencoding object:

"encoding": {

"x": {

"field": "state",

"title": "State Name",

"type": "nominal",

"sort": {

"field": "gwh",

"op": "sum",

"order": "descending"

},

"scale": {

"type": "linear",

"zero": true

}

},

"y": {

"field": "gwh",

"title": "Gigawatt hours (GWh)",

"type": "quantitative",

"scale": {

"type": "linear",

"zero": true

}

}

}

And our final chart:

This tutorial provided some basic styling functionality by accessing the Vega Lite editor in Chart Builder directly, but it can do so much more. Check out the official Vega Lite examples, tutorials, and for an in depth look.

One of the most powerful ways to share results of your data analysis is through visualizations. On data.world you have access to both integrations formany third-party tools and also to Chart Builder, a visual editor for Vega-Lite built specifically for data.world. It is the perfect tool for those looking to create a simple visualization that is lightweight and easy to embed.

Visualizations in Chart Builder can be made from either a file or a query. They can be saved in various formats, shared, or embedded in the various Simple Editor windows.

In this article we'll explore:

Using Chart Builder with files

Using Chart Builder with queries

Formatting options for visualizations

Charting more than one column in graph

Sharing and saving visualizations

Troubleshooting

Using Chart Builder with files

You can create visualizations of tabular data files in either projects or datasets without ever writing a query. To follow along with this example, open the dataset Shark attack data updated daily. To create a visualization of all the data in the file, select the Viewicon to the right of the file name on the overview page:

here

You'll be taken to the data workspace and a view of all the data in the file. Next, click the dropdown arrow next to Open in appand choose Open with Chart Builder:

If you're using Chart Builder for the first time, you'll be directed to a page requesting authorization. After authorizing, you'll be redirected to the Chart Builder workspace.

Chart Builder comes with two options for creating and modifying charts: a Visual Builder and a Vega-LiteEditor. The easiest way to use it is to create your initial chart on theVisual Buildertab and then switch over to theVega-LiteEditorto make any changes outside the scope of theVisual Builder. See our article on using Vega-Liite or the Vega website for more information.

To create a quick bar chart of the number of people in the dataset who have been fatally attacked by sharks, select the fieldfatal_y_n from the dropdown list for the X axis, andCOUNT(*) from the bottom of the list on the Y axis. Like magic, our chart appears on the right side of the screen:

Using Chart Builder with queries

Chart Builder can also be used on queries. Using a query as the basis for your chart enables you to:

Clean up data (e.g., remove NULL values)

Filter out data (e.g., specify a time period, a specific value, an aggregation, etc.)

Change your data structure so that it can be charted

In the shark attack dataset referenced above there is a saved query called Query for analysis by year or country This query has been written to exclude NULL values and remove non-binary entries on specific fields. It can be used to create a number of different charts. Click the dropdown arrow next to Open in appand choose Open with Chart Builder:

Formatting options for visualizations

Continuing with the example above, to make a chart with circles for marks that compares the number of attacks on men vs on women across the years select Circle from the Marks dropdown, and year from the X axis dropdown:

If the axis doesn't display the way you want it to, you can override the default format for the Type under the Options dropdown. In this case the year was read as a number because of the underlying data type and the type was set to Quantitative when Ordinal was the right choice:

Set the Y axis to COUNT(*) and Color to sex:

If your chart is not appropriately sized for viewing you can manually set the chart size. A size that shows the data of this chart to best effect is a width of 950 and a height of 730:

Another handy thing you can do in the options section is order the results in your graph. You may have ordered them already in your query results, but that order does not carry over to the graph. For this example we'll use the saved query Countries with >10 unprovoked attacks since 1960 with mortality data. After running the query, click Chart, set the X axis to country and the Y axis to # attacks:

To sort by the country with he most attacks select Options on the X axis, chooseDescendingfor the sort andy - # attackson the field:

To add information on if the attacks were fatal or not, selectfatal_y_n next to color.

If your results do not seem to display as they should, check to make sure the field you are sorting on is not being improperly aggregated. Being able to switch over to the Vega-Lite Editor is very handy for identifying this kind of configuration issues. Looking in the Vega-Lite Editor in the area dealing with "sort" the operation is set to "average":

In this case "sum" is the correct option, and upon replacing "average" with "sum" the visualization displays properly:

NOTE: Once you have made edits in theVega-Lite Editor you can no longer make any changes in theVisual Builder so save yourVega-Lite Editor changes for when you are finished building the chart with theVisual Builder.

Charting more than one column in graph

It is occasionally the case that you'll want to include the data from more than one column in a chart. In some cases the values in two columns are related and could have been stored in one column but weren't, e.g., separate columns for Male/Female, high-temp/low-temp, max value/min value, start date/end date, etc. In other cases the data in the columns isn't related so you wouldn't expect it to be stored in the same column, but you still want to use it all together in the same graph. It is possible to view the data from multiple columns in Chart Builder, it just requires modifying your original query with UNPIVOT so that all the columns you would like to view are combined into one column in your query results. Detailed instructions can be found in our article Using multiple columns of data in Chart Builder visualizations.

Saving and sharing visualizations

Chart-builder visualizations on data.world can be saved in a number of json, image, and html formats shown under the Download button:

There are also a variety of options for sharing your visualization on data.world:

SelectingShare > Insight lets you add your visualization to any project on data.world for which you have permission:

To share the insight you are prompted to chose a project where you will share it, to give it a title, and optionally to add comments. The final option (selected by default) is to save the visualization as a Vega-Lite source file on the project:

Share > File lets you add the visualization to any dataset or project for which you have permission.Share > Markdown Embed (Comment)you can embed your chart in any place which uses Markdown (e.g., insights, comments, summaries). By default the embedded chart will be a static rendering of the data from when the visualization was created. However using theVega-Lite Editoryou can create a 'live' chart that updates as the data on which it's based updates. The shark attack dataset is an example of continually-updating data.

To make a chart 'live', go to theVega-Lite Editorand scroll down to the section referencing the "data" parameters:

Under the "data" element replace "source" with "url" and add a hardcoded url for the query that drives the visualization (you get this in the workspace while viewing the query), and add a "format" element with the type "csv":

Then when you selectShare > Markdown Embed (Comment), you'll get Markdown text for a live version of the visualization that you can copy and paste into Insights, Comments, and Summaries on data.world:

Here is an example of the live visualization above used as an insight on a project that uses the dataset:

Finally, if you want to share a link to the Chart Builder screen for the visualization so someone else can edit and run it, you can do so with theShare > URL option:

This example can also be found on the discussions page for the dataset. Notice the use of Markdown stying to make the comment more readable. Detailed instructions for using Markdown on data.world can be found .

Troubleshooting

Error loading data

An expired token can cause one to receive a"Error loading data." message when opening Chart Builder. To remedy this:

Click on your account avatar on the top right corner of data.world and go to 'Your integrations'

Select the Chart Builder tile

On the Chart Builder page, select theManage tab

Click theRevoke button and disconnect the Chart Builder integration

Click theEnable integrationbutton and authorize access

Re-launching Chart Builder will now allow it to fetch the data successfully

Blank chart

When using the Vega-Lite editor to modify the Chart Builder output, many errors cause a blank chart to display. Troubleshooting must be manually carried out in this case - the Vega-Lite editor does not include any error identification functionality.

With data.world's Simple Editor you can quickly and easily add different kinds of files from various sources intoproject and dataset summaries, posts, and insights. The articles Intro to the Simple Data Documentation Editor and Using the Simple Editor command menu describe the editor and the features of the command menu. In this article we will cover:

Using drag and drop to embed assets

Various sources for embeddable assets

How different file types are embedded

Embedding assets vs. linking assets

Using drag and drop to embed assets

In addition to using the command menu, you can also embed assets into a Simple Editor window with drag and drop. Drag and drop works with assets located in the current project or dataset, on your local drive, a networked or cloud drive, and a web page. To embed an asset by dragging and dropping it the asset needs to be dragged to the beginning of a new line in the Simple Editor window. Assets placed in-line render as links rather

Various sources for embeddable assets

The command menu is useful for embedding assets from the current dataset or project into the editor window, but assets in connected datasets or located outside of data.world must be embedded with drag and drop. Sources for embeddable assets include:

The current project or dataset

Desktop or cloud storage

From a URL

The current project or dataset

While recently modified files in the current project will show up in the command menu, files in linked datasets do not. Drag and drop works for both the current project or dataset files and also for the connected dataset files. To include any asset from the left sidebar of the workspace select it and drag it to the beginning of a new line in the editor window:

Supported file types

The files in your project or dataset that can be viewed in the workspace typically render in full or as a preview in the Simple Editor window. PDF's are the current exception to this rule and can only be embedded as a link. Tabular files are displayed as a preview with a link to the full file. Text files (.txt, not rich-text files--.rtf) and most image files are also previewed in the editor. See the article for detailed information on the rendering of specific file types.

Queries and insights are both rendered as previews in the Simple Editor. Notice that for queries there is also an option to run the query in the bottom right corner of the window:

If you select the Run query option you'll be taken to a new tab in the workspace which will display the full results of the query (or the maximum number allowed for larger tables):

To get back to the summary window click on its tab at the top of the workspace:

Parameterized queries can also be embedded into the Simple Editor where they will be executable by the reader. The query is previewed in the editor window with all the parameter fields showing the default values. The reader can replace the defaults with other values and interactively run the query without ever leaving the summary, post, or insight. Running the query returns a preview of the first five rows of the query results. ClickingSee all runs the query in a workspace and returns all the results from the parameters set by the reader (up to 10,000):

Desktop or cloud storage

Embedding files from your desktop or a cloud storage device like iCloud, Dropbox, or Box is easy with the editor's drag and drop interface. Simply choose the file and drag it to the beginning of a line in the Simple Editor window. Non-image files will be added to the dataset or project:

You will then need to embed the uploaded file into the editor window either with the command menu or by dragging and dropping it. If the file you choose is a supported image file, the imagewill be added to the editor window without being added to the dataset or project as a new file:

From a URL

Another source for embeddable content is a webpage. URL's can be embedded into the beginning of a line in a Simple Editor window and they will show up as either a preview of the webpage or as a link to it, depending on how the source page was configured by its author:

Embedding assets vs. linking assets

With the Simple Editor you have the choice of either embedding content directly into an editor window or linking to it. As we have seen, embedding content is done by dragging an asset to a new line in the editor window and dropping it into place. If, instead of dropping an asset at the beginning of a new line, you insert it in-line, a link to the asset will be created.

Here are two contrasting examples of embedding and linking an asset:

Parameterized query embed and link

Image embed and link

In the article Intro to the Simple Data Documentation Editor we introduced data.world's Simple Editor that allows you to quickly and easily add different kinds of files from various sources intoproject and dataset summaries, posts, and insights. In this article we will cover the command menu options:

Recently updated assets

Checklists

Code blocks

Embed Content

The command menu (accessed by typing forward slash ( / ) at the beginning of a line in the editor window) contains a list ofitems you can embed including recent files, queries, insights, and various content block options.If your cursor is not at the beginning of a line when you type the forward slash, your content will be linked, not embedded. More information about embedding vs. linking can be found in the article Embedding assets with the Simple Editor.

Recently updated assets

The first several items on the command menu are files, queries, insights, and other assets specific to your current dataset or project that you have recently modified:

Embedding assets with the Simple Editor

If the item you wish to embed is not in the recent items list you can begin typing its name and autocomplete will take over:

Checklists

Creating a checklist is as easy as selecting the item on the menu and then entering all your list items separated by a return. Checklists format as items in a list and the boxes can be checked during the edit process. Once the summary or insight has been saved, the state of the checkboxes is set and can be changed by opening and editing the summary or insight again:

Code blocks

Selecting< > Codefrom the dropdown inserts a formatted code block where you can type or paste a code snippet:

Embed Content

The</> Embed Content menu item inserts the formatting required for a URL so all you need to do is paste the URL into it and it will be embedded after it's saved:

When adding embedded content, the embed link must be the only text on its line. You cannot, for instance, embedded content in the middle of a sentence.

More information on embedding and linking content can be found in the article

The Simple Data Documentation Editor--Simple Editor for short--is data.world's built-in WYSIWYG editor for project and dataset summaries, posts, and insights. The Simple Editor generates powerful data-enabled markdown, and allows you to create data-rich documentation for your datasets and projects faster than ever before. Now you can:

Use drag and drop or autofill to embed or link assets in various formats from many different sources

Benefit from easy-to-use styling or standard Markdown

Insert a variety of content blocks

Use Markdown shortcuts while in the Simple Editor

Switch back and forth between the Simple Editor and Markdown

in this article we will cover

Accessing the command menu

Embedding @mentions to other users and organizations

Formatting options and shortcuts

Switching between the Simple Editor and Markdown

Accessing the command menu

When you place your cursor on a new line in a Simple Editor field, a list of shortcuts for basic embed and format options appears:

Using the Simple Editor command menu

If your cursor is not at the beginning of the line your content will be linked instead of embedded. More information about embedding vs. linking can be found in the article Embedding assets with the Simple Editor.

The forward slash (/) brings up the command menu which is covered in detail in the article Using the Simple Editor command menu.

Embedding links to other users and organizations

The ampersand ( @ ) brings up a list of users and organizations with whom you've recently interacted in order to @mention them. If the person or org you want to tag is not in the list, you can begin typing the name for a list of options:

The @mention takes readers to the profile page of the user or organization who was tagged.

Formatting options and shortcuts

The last item on the initial prompt of commands for the Simple Editor is highlight text. It serves as a reminder that when you are in the Simple Editor you can highlight a section of text to get a menu of the formatting options built in to the editor. :

The options are Bold,Italics, code, header types 1-3, and links. In addition to formatting with the highlight text menu you can also use the basic keyboard shortcuts like +b for bold,+ifor italics(Ctrl+ for Windows). The standard Markdown shortcuts (# Header1, ## Header2, **bold**, _italics_, etc.) can also be used in the Simple Editor window. For more information on Markdown styling see our Markdown syntax reference.

Switching between the Simple Editor and Markdown

Though the simplicity of the WYSIWYG interface of the Simple Editor will appeal to most users, those who prefer to use Markdown can switch to Markdown by selecting the Switch to Markdown link at the top of the edit window:

It is also possible to do some editing in the Simple Editor and continue editing in Markdown or vice versa. Both versions are maintained simultaneously:

More information about how the Simple Editor works can be found in the article.

This article is a quick introduction to the new project workspace layout. More general introductory information on data.world can be found in the Getting started guide. If you have been using data.world for awhile and would like information specifically about the changes between the old and new workflows, see our article reviewing the changes.

Page Layout

When you land on a project workspace you can tell what's showing in the main area of the screen by what'shighlighted in the Project directory in the left sidebar:

Supported file types

The project workspace has five main parts:

The left sidebar is the Project directory

Underneath the header in the center are the tabs and actions for the current object

Beneath the tabs In the center is the object viewer

In the right sidebar (when visible) on the top is the About section with information about the currently selected object

For some objects, the bottom of the right sidebar contains the Project schema

Project Directory

The Project directory is the navigation area of the workspace. At the top is the + Add dropdown which is where you can add files to the project, link datasets to it, add posts or insights, and add new SQL or SPARQL queries.

Below the+ Add buttonare a link to the Home tab, and the main project files (Project summary and Data dictionary). Information about the project summary and the data dictionary can be found in the articles Description and summary, and Data dictionary.

The next section of the Project directory is for project files. Project files are any data resources uploaded or saved directly to the Project--not to a dataset.

Project files cannot be added to other projects or datasets. They are only available in their project. See the article datasets and projects for more information.

Below the project files are connected datasets. The datasets used in this project may also be used in and linked to other projects. When there are changes to the underlying dataset, all projects using the dataset also update. All files and queries associated with a dataset are linked to the project and can be used in it:

More information on how to create a dataset is in the article Creating a dataset.

The last two sections of the project directory,queries and insights, are also specific to the project. The Queries section contains all the stored queries in the project. Learn how to write and use queries in our article Query basics. We also have a SQL tutorial and a SPARQL tutorial to help you learn or improve your query language skills.

Insights arefindings, conclusions, or interesting points for discussion about your project. Theyallow you to capture conclusions from your work, packaging them up in a way that quickly communicates a nugget of information, while giving the viewer the tools they need to dig down into your methods and sources. See Posting insights for details on insights.

Tabs and actions

Whenever you select an object or the + Addbutton from the Project directory, a new tab opens up under the header bar with a preview of that object. Underneath the tabs are an icon indicating what type of object is shown (the name of the object shown and to the right of the name are any actions that can be taken with that object. and whether it is shared with all users of the project or private only to you. Further to the right is a link to create a new query template, or parameterized query. You can learn more about parameterized queries from the article Using query templates. The last arrow icon is used to either expand or collapse the about and project schema panel. For more information about query-specific actions, see the article Query basics.

Object viewer

The central are of the workspace is devoted to to viewing or interacting with the object selected in the project directory. The viewer renders previews of most file types (for a complete list of files supported by preview see the article . When the object is a query, the viewer is split into two parts: the query editor at the top and the query results at the bottom. The query editor is where you compose and run queries against your data. In addition to typing in your text, data.world's query editor also provides auto-complete and auto-format of commands, columns and and tables:

The bottom center of the screen is used to display results of queries or error messages when queries are not written correctly:

About

In the upper right corner of the project workspace is a pane with the information about the object highlighted in the project directory. It is used for queries and files to give you more information about them that might be relevant for their use:

Project schema

The project schema shows up at the bottom of the right sidebar whenever the object displayed in the main section is a query. It contains a list of the queryable entities in the project (datasets, tables, and columns). Items can be expanded or collapsed, and selecting either a column or table copies its name to the clipboard for easily pasting into the query editor:

Easily pull data directly into Tableau to create your visualizations using data.world's web data connector. By using the connector as your data source, Tableau can even keep the data in sync with data.world to ensure your visualizations are using the most accurate data. Follow the instructions below, or checkout our video tutorial here: How to pull data into Tableau

To use the connector from Tableau or Tableau Public v.10+, select the Web Data Connector option under Connect > To a Server within your Tableau application:

contact us

In the window that opens, we're going to enter the web data connector URL. We'll walk through getting that URL in the next step...

Switch back to data.world and on the dataset, data file, Data Project, or query you'd like to import to Tableau, select Export > Tableau and copy the connector URL from the resulting window:

Returning to Tableau, paste the connector URL into the input box. Note that when you enter the URL, the input box disappears and moves to the URL field so you'll just need to hit Enter on your keyboard to load the next page.

By using the connector URL from the 'Export > Tableau' menu, the data.world web data connector will be pre-populated with your dataset URL or query details. Note that the first time you use the connector, you'll be prompted to log into your data.world account:

That's it! Click Submit and the connector will grab all of the dataset tables, columns, and data types for that dataset and pull them into Tableau. If you'd like to bring in another dataset, you can go to Data > New Data Source and repeat the steps above.

Once you've created and published your viz to Tableau Public, use the published URL to share it back on data.world as an Insight on a Data Project, as a discussion post, or within your dataset or Data Project summary using the markdown syntax @(https://public.tableau.com/shared/url):

Here's an example Tableau embed with the corresponding markdown.

More on queries:

Note that we've recently added the ability to pull in a specific query via the web data connector. This is great for when you're working with really large files that may take a while to import via the connector, but you really only need a subset of the data. To import a query, just click on the Export > Tableau option from the query view in your workspace on your dataset or Data Project:

Again, copy the web connector URL from the popup:

And paste that in as the connector URL from Connect > Web Data Connector in Tableau and press enter:

See that we're passing in the dataset, query type (SQL or SPARQL), and the query you defined. Click submit to pull in the query results to Tableau.

As always, if you have any feedback or run into any unexpected issues with the connector.

One of the most powerful ways to work with data on data.world is to do it without ever leaving your favorite application. There are already many applications which have data.world integrations built for them, and more are constantly being added. For a comprehensive list of the current integrations available see our Integrations page. This article covers using our integration for one of the most common data analysis tools: Microsoft Excel. This article was written on a Mac using Microsoft Excel 2016 Desktop for Mac so the screens shown may vary from yours if you are using Excel for Windows.

Installing the data.world add-in to Excel

To start using Excel with data.world you need to install the add-in in Excel, and then enable the integration in data.world. You can get the add-in on the Microsoft app store page. On the App store page, enter data.world into the search bar and click Add:

Excel integration

Once the add-in is installed you'll have three new items on the right side of the home tab on your Excel menu bar:

When you have the add-in installed, go to the Excel integration page on data.world and select Enable integration:

Authentication

To use Excel with data.world you'll need to either sign-in to your data.world account or get an access code/API token from within data.world. Here are instructions for both.*

To use your data.world account, select one of the data.world icons from the top right of your Excel screen. When the login page appears, select Sign In, and enter your data.world credentials. To use the access code or an API token, selectaccess code on the login page:

You can get an access code from the page under theManagetab by selecting Get token:

Copy the token and paste it into the window linked from access codeon the login page.

Uploading Data From Excel to data.world

To upload data from Excel to data.world, select the Sync Data button on the right side of the Home tab of the Excel menu bar. You can either upload a selection from the current sheet or the entire current sheet. Because the worksheets in an Excel file are stored as individual files on data.world, if your Excel file has multiple worksheets you will need to upload them one at a time:

When you upload directly from Excel you have the option to add the file to a new dataset, to an existing dataset, or to an existing project:

Note: If you choose to add the spreadsheet to a project it will not be available for use in any other projects or datasets.

If you upload to a new dataset you'll also be prompted to set the permissions for it. The options are private and public--you cannot set permissions at the organization level from within Excel. However you can adjust permissions to the dataset from within data.world after the dataset has been created:

Updating data.world data from within Excel

After you have uploaded a spreadsheet to data.world you can continue to work on it in Excel and then sync your changes when you're done. To update an existing dataset selectSync data, and click the upload button to the right of the dataset name:

Importing data from data.world into Excel

If you're working in Excel and you want to import a file from data.world into your current workbook, select Import data from the Excel menu bar and choose either a new dataset or one you have previously worked with in Excel:

If you choose to import a new item you'll be prompted to choose the source dataset or project from all of the datasets and projects you explicitly have permissions to (you do not get a list of all the datasets and projects on data.world--just the ones you own, that are owned by an organization you are in, or which have been shared with you). Next you choose whether to import a query or a table (a table is any tabular file in the dataset or project you chose), and if you want to import it to a new tab in your workbook or to the current sheet:

Note: if you import into the current sheet, all the data in the sheet will be overwritten.

After working with the data in Excel you can sync it back up to data.world.

Publishing insights from Excel to data.world

While data.world has a a built-in Chart builder application and an integration with Tableau, if you prefer to work on your visual data analysis in Excel you can still upload your charts as insights to projects on data.world. To upload a chart from Excel choose the Publish insights menu item and select your chart:

You'll be prompted to select an existing project or to create a new one, and if you choose New project you'll set the permissions for it to public or private. After uploading, your chart is available on the insights tab of the project along with any comments you added:

*The only time it matters which you use is if your organization uses SAML for authentication. SAML users must use theaccess code/API token.

Slack is a powerful communication tool for teams and when paired with data.world it extends your control over your information flow in dramatic ways. Integrating data.world with Slack enables you to get real-time updates in Slack whenever changes are made to data.world accounts, datasets, and projects--keeping you and your team constantly in the loop.

With Slack and data.world you can:

Easily subscribe to projects, datasets, and accounts to receive notifications when updates are made.

Share rich messages when you link to data.world so your teammates won’t be left in the dark.

Preview query details and easily link to a specific query with one click.

Quickly view your subscriptions, unsubscribe, and explore further through inline commands.

In the next sections we'll show you how to do all of these things and more!

data.world's Slack integration is not created by, affiliated with, or supported by Slack Technologies, Inc.

Installation

To install the data.world Slack integration, select Add to Slack button at the top of the Slack Integrations page:

You'll be prompted to either enter your Slack workspace URL (you must be an administrator for the workspace to install the app) or to create a new workspace:

After you enter your URL and select Continue you'll be taken to the final set-up screen that tells you how data.world will interact with your workspace:

Once the data.world app has been installed on Slack it will show up on the bottom left of your Slack workspace in the Apps section. The first message from the app will contain a welcome message with instructions on how to use the app and where to get help:

Subscription

In the example above we have a channel called crm for our customer relationship management team. If they want to get updates whenever changes are made to the crm datasets, we would need to set-up their channel as follows:

Add the data.world app to the channel - this needs to be done for every channel ion the workspace that wants to use the app.

Subscribe the channel to the projects, datasets, and accounts for which the members should get updates.

To add the data.world app to the channel, select the channel from the list in the left sidebar and enter the command/invite @data.world:

Once the app has been added to the channel you can set up subscriptions to projects, datasets, and accounts. To subscribe the crm channel toCRM Project, enter the command /data.world subscribe https://data.world/siyeh/crm-projectinto the command line of the channel:

Notification

After a channel has been subscribed, whenever an update is made to the subscribed item (project, dataset, or account) everyone in the channel will automatically get a notification of the update with a link to it:

In the example above the file can be commented on in the Slack channel or on data.world by selecting the Discuss button at the bottom of the message:

Rich messages

With the data.world app for Slack installed whenever you reference something on data.world in your channel you get a rich text preview with embedded links and the option of subscribing to the item (if a project, dataset, or account) from the message:

Preview queries

Another powerful feature of the data.world app is that you can paste the URL of a data.world query into Slack and it gets rendered into preview and has links for others to access directly without having to hunt around the project or dataset to find the query:

View and manage subscriptions

Managing your subscriptions and getting help for commands is easy too! To see what a particular channel is subscribed to, enter the command /data.world list into the channel dialog and you will get back a list of all the projects, datasets, organizations, and users the channel is subscribed to with links to each of them and the name of the person who subscribed to them:

The correct command format for subscribing is stored in Slack in the first message from the data.world app so you'll always have it readily at hand. The syntax for all commands also can be accessed by typing /data.world help:

Redirect to https://docs.data.world/tutorials/sparql/

When a dataset contains many files, determining the purpose of each file can be difficult without examining it more thoroughly. By addingfile labels, you can see each file's category at a glance. The following file labels are available:

raw data

clean data

documentation

script

visualization

report

You can add file labels from a dataset overview page by clicking the three dots icon on the right side of the file preview and choosing Edit file metadata:

That will bring up a new window where you can add file labels and edit other file metadata:

To update those labels later, just follow the same steps and add or remove them as needed.

If you're looking for search syntax and terminology see our article on advanced search.

In this article we'll cover basic information on using the search including:

Using the search bar

Understanding search results

Finding related data

On data.world you can not only search for many different kinds of assets, but you can also filter your results by resource type or integrated facets like status, owner, or tag. Each refinement of the results set tells you how many assets meet the combined criteria, and it's easy to drill down through the myriad results to find just the asset you are looking for. See the article on filtering search results for more information on narrowing down your search results.

Using the search bar

Sometimes you just want to get back to the last thing you viewed. A quick way to do that is to click in the search bar. There you'll see a dropdown with the last three resources (people, datasets, or projects) you accessed:

evaluating search results

There is anautocomplete component to the search bar so as you type you see the most relevant people, datasets and projects for the string you are entering:

The search bar is where you can go to find a variety of assets across data.world. From it you can search for not only datasets and projects, but for people, organizations, data analysis, queries, specific resource types, and metadata. Places searched include:

titles

summaries

descriptions

tags

insights

column names

creator or owner

creation or modification dates

metadata

Besides searching for text anywhere in the data, you can also search the metadata for matches on creator, owner, tags, or modified date. For example, if you wanted to find all datasets owned by the user @us-usda-gov you could search owner:@us-usda-gov :

You can also use the search bar to search for multiple search terms. When you enter more than one text string into the search bar you will get results that contain all the strings in them.You can search for an exact match on a text string by enclosing the string in double quotes E.g., a search on "honey bee" returns the following:

Finally, you can combine operators to broaden your search results. E.g., if you wanted everything that had either cat or dog in it you could run the following search:

For all the ins and outs on using the search bar and a complete list of all the operators you can use, see the article on advanced search terms.

Understanding search results

After you run a search your results are presented in a list of resources with detailed information about each of them. In the example below, the search term was "bee". Below the search bar you can see that the search returned 325 results associated with me or with an organization to which I belong. If I wanted to expand the results set, I could check the box on the upper right to include results from the rest of the community (723 total). If you are not in any organizations, there won't be an option to include community results because you will already see all the community results. Alternatively I could limit my results to a particular resource type by selecting the dropdown menu next to All resource types.

Each item in the results list is shown on a resource card that contains an appropriate set of information and actions for that resource type. In the list above, the first result was a person with the username @bee. On that result card is a link to the user's profile and a link to follow the user. The next result card is for a dataset:

At a glance you can see that there is a lot more information on this card including:

An icon for the type of resource it is (see our guide to icons for details on all the resource icons on data.world)

The title, creator, and owner

Details on when it was last updated

Your role in the dataset or project (if applicable)

The description of the resource

How many projects it's used in (for datasets)

How many linked datasets it has (for projects)

How many files and tables it contains

The tags used on it

How many people use or follow the resource(bookmarks)

A link to the discussions page for it

The additional information on your search results can be useful for determining whether a resource is a good fit for your project. Format, source or provenance,recency, and frequency of citation or popularity are all useful metrics for evaluating data. See our article on for details.

Finding related data

You can also find information about other related datasets from the overview page of a dataset including:

Other resources with the same creator (right sidebar)

Other resources with the same owner (top left)

Resources with the same tags

Related projects

Clicking through to any one of them will take you to another set of data that might be relevant to your current project or that you might be interested in looking at later.Finally at the bottom of the overview page there is a set of curated suggestions of related datasets:

Once you have created a dataset or project and added your files to it, you can make it easier to find and more useful to others by describing, or documenting it. Documenting consists of creating the metadata for your dataset or project and helps others to trust your data and work. Searches on data.world also look at titles, descriptions, summary, and tags to match search strings so the more completely you describe your data the more chance it has of being found. The metadata collected is:

Description

Summary

Tags

Data dictionary

Licensing

The starting point for describing your data is the dataset or project overview page. From here you can edit the description, create the summary, assign tags, set the licensing, and complete the data dictionary:

Description and summary

Dig in further to each element via the links above, or continue to.

Introduction

data.world taps into more of your team’s collective brainpower to achieve anything with data, faster. Modern data teamwork means everyone from data scientists to domain experts contributes value to data projects in the form of raw data, expertise, or simply by asking great questions and connecting the right people to help. Some of the benefits of our platform are that it:

Makes data easier to find -You can collect data from across scattered or disconnected sources, add context to make it more discoverable (with search and your homepage feed ), and use it immediately. Plus, yourlibrary of datasets, projects,and bookmarks help you quickly get back to the data you use most.

Makes data easier to understand -Context travels right alongside your data so everyone can immediately understand it and use it to power better decisions. Amazing data previews, data dictionaries, and other perks keep context close at hand while enabling Q&A and collaboration via discussions.

Enables teams to work together faster -You can stop emailing spreadsheets! data.world helps more people from your team securely and effectively work together on solving problems with data, whether they’re technical or not. For those of you that are technical, you’ll love this: you can write queries against any data without standing up a database, connect it to the tools you already know and love, and share the results of your work all in one spot, all without missing a beat.

Puts data and insights in the hands of those who need it -With data.world you can keep valuable work from getting lost in inboxes or ad hoc conversations and give everyone in your organization insights they can usealong withthe data behind them. As a result, your analysis and work will be more reproducible, reusable, and available for further collaboration.

To jump right in and get your feet wet take a look at our data.world tutorial!

Contents

Welcome to data.world!

Connecting your data

Verifying your data

Documenting your data

Extending your data

Adding contributors

Working with data

Querying and analyzing

Visualizing data

Sharing and collaboration

Governance and auditability

Additional resources

Welcome to data.world!

At data.world, we want to tear down the barriers between people and data by building the most meaningful, collaborative, and abundant data resource in the world. We're building a platform where people can:

Discover, prepare, and share relevant data from a wide range of sources

Reduce the overhead of managing numerous file formats from many different places

Explore and create a shared understanding of your data

Contribute and discuss data in one place to aid fast, efficient, and fruitful collaboration.

At data.world we offer organizations and teams private, secure environments in which they can collaborate together. In using data.world, you and your team can put data and insights in the hands of those who need them while keeping valuable work from getting lost in inboxes or ad hoc conversations. As a result, your team's analysis and work will be more reproducible, reusable, and available for further collaboration.

data.world supports the largest open data community in the world, where we’re excited to connect members with a vast collection of scientific research, government, and demographic data, as well as other members who are interested in similar data work so they can join forces to solve real problems faster.

Learn more about working on data.world in this video:

Regardless if you’re working in an open or closed setting, data on data.world can be brought together in one of two ways:

Datasets help make your data accessible, reusable, and understandable. They contain the data and any additional metadata, scripts, or files that are related to that data. Datasets are the building blocks of data projects.

Projects help organize your data, documentation, and output in one place when working on a specific question, project, or analysis to create reproducible work outside of the black box.

In this guide, we’ll walk through the basics of data.world, covering features and functionality to take you through your first data project from start to finish. Before getting started, ensure you’ve setup your account and joined any organizations you’ve been invited to,and next we’ll walk through finding and adding data.

Connecting your data

You can find thousands of free, open data resources on data.world to work with (check out our guide to finding data ), but quite often you’re going to bring in your own data to use.

When adding data to data.world, you’ll typically want to create a dataset. A dataset is simply a repository of data including data files and associated metadata, documentation, scripts, and any other supporting assets that should be stored alongside the data. Datasets can be created manually, which is what we’ll walk through here, and we also have automated options for working with larger datastores ( contact us for more details).

To create a dataset, click on the + New dropdown on the right side of the header bar and select New dataset:

Distinct values

After you've opened the new dataset you will:

Add a title (up to 60 characters)

Select an owner (if prompted) - if your dataset is for an organization, we recommend creating it under the team account to keep your organization's work within a single library.

Choose the visibility of your dataset:

Private - only accessible to you,

Organization - allows your team to view the project

Open - available to the data.world community

If you’re unsure which permissions to choose, we recommend starting private and adding contributors and increasing visibility as you go.

Next, either drag and drop one or more files into the add data box, select the Add data button for additional source options,or save your dataset and add data at a later time.

There are multiple ways to add data and connect your data sources:

Upload from your computer, or select cloud storage services (Box, Dropbox, Google Drive)

Pull directly from URL or API (see the article Adding files from a URL for details)

Connect your data via an integration (check out our super connectors if you don't see your datastore listed)

To upload files from your computer either drag and drop the file(s) from your hard drive to the Add data window or select the Add data button for more options:

So that data owners can fully document their data, data.world supports all file types. Use a script to clean the raw data? Upload it so others can see and build off your work. See What file types can I upload? for more information about how data.world handles different file formats, and when you're ready, select the Create dataset button at the bottom of the page to continue.

Verify your data

When you upload data files (csv, xlsx, json, etc.) to data.world, behind the scenes we're actually converting any readable data to a graph database, so that, regardless of format, it becomes instantly previewable in a tabular format as well as queryable within data.world. More on that later, but due to how we process these files, we can also infer data types, high-level stats about the data, and any potential issues in the data.

After data.world processes your data file, the data inspector will automatically attempt to identify and display any potential issues including blank cells, duplicate rows, and numeric values or string lengths far outside the standard deviation for their field.

The number of potential issues is flagged by the orange exclamation mark at the top of the data preview:

Select the link to open the data inspector and then review the suggestions. If you feel the highlighted issues are not relevant select Dismiss to prevent them from showing again:

Any changes to the data will still need to be made in your local copy and reuploaded, where we'll again process the file and report on any potential issues identified. As long as you upload a file with the exact same name, the previous version will be overwritten (although all versions are maintained and accessible!).

This is also the time for you to verify your data to ensure that the columns in your tables were assigned the correct formats on upload. The format of each column is shown to the left of its name in the table. If you click on the icon you will see a pop-up with more information on the field:

If you wish to change any column formats and also jump into describing your data, select Edit in the top right of the pop-up which will take you to the data dictionary.

Documenting your data

Once you are confident that your data is accurate you are ready to document it.The more complete your documentation, the more useful and understandable your data will be when it comes time for you or others to use it.

The basic information about the entire dataset can be accessed by selecting the Settings tab or by clicking Editfrom your dataset main page:

Within the settings, you can:

Add a short, meaningful description to summarize your data

Add tags (individually or comma-separated)

Select thelicense type for your dataset(see Common license types for datasets for more details)

Edit the visibility level

Manage auto-sync functions: if files are connected via a URL,choose to automaticallyrefresh hourly, daily, or weekly

Select how to import compressed files

Delete the dataset

In addition to the metadata under the settings tab there are two more ways to document your dataset: the summary and the data dictionary. Both are accessed from the information box at the top of the overview page:

Use the Summary section to tell your data's story:

Where did the data come from? Cite and link to your sources or include details for a 'citation request'. Not only does this give "credit where credit is due", it helps other members evaluate the data's suitability for their needs.

If you think a particular piece of context will be useful to other members in understanding the data, add it.

The best summaries cover the “who, what, where, when, why, and how” of the data.

Make it visually friendly with Markdown styling. It's easy to learn and goes a long way.

Back on the dataset main page, click View next to the data dictionary to further document the files and columns within a dataset. You can add descriptions and labels to the individual files, change column types, and add column descriptions:

Select + Add a description under a filename to add a description and labels to the file:

Within the above file view, select Column details to add detailed descriptions for each of the columns, particularly if your column names are obscure. You can also specify the data type:

Extend your data

After uploading data to data.world, if your data contains a common, general type of information that data.world recognizes, we will suggest related information that you can use to enhance your data and research with. We call this data matching, and you'll see it in the platform for things like postal codes, state abbreviations, and medical codes (see this query for a full list), but it could also be setup for identifying data specific to your organization such as product or customer ids to more quickly pull supplemental data into your dataset.

If you see a green triangle on any columns after adding data to the platform, you know we've found a potential match:

Clicking the corner will present a menu which where you can jump in to review the matches data.world has identified and confirm whether the data in youroriginal column matches the type data.world has identified.

If you confirm a match by selecting Add matched column, data.world will display any related data that could also be pulled in due to your confirmed match:

Note that adding matched or related fields will not change your underlying dataset and you can un-match them at anytime. Findmore details at Match and extend your data.

Adding contributors

Once you're ready to invite additional people or teams to your dataset to contribute to or view it, select Contributors from the dataset page and then click the Invite additional contributors link:

You can search for and invite other data.world users by name or username or you can invite external users by entering their email address and sending them an invitation to join the platform. For organizations, you can quickly add your entire team by adding your organization as a contributor:

There are 3 types of permission you can assign to contributors: View, Edit, or Manage;each providing a different level of control over the dataset:

Now that you've added your data to the platform and allowed your team access to it, lets move on to working with the data on data.world!

Working with data

Once your data is added to the platform, or you've utilized the robust search capabilities to find and discover open or team-specific datasets you have access to, you'll probably want to do something with them! This is where projects come into play.

While you can do much of the same things in a dataset, the real place to work with data on data.world is in a project. Projects allow you to connect to data from various datasets to combine and analyze them, document your analysis, post insights, share with others, and collaborate through discussions.

To kick off your project, select the + Add link on the top right of your screen and choose New project:

From there you will be taken to a basic Create a new project page where you'll configure:

Owner (if prompted): if your project is for an organization, we recommend creating it under the team account to keep your organization's work within a single library.

Project name

Project objective: projects are best when they start with a clear question or goal.

Project permissions:

Private, - only accessible to you.

Organization - allows your team to view the project.

Open - available to the data.world community.

If you’re unsure which permissions to choose, we recommend starting private and adding contributors and publishing out as you're ready.

Once your project has been created, add or link data to your project via:

Finding and linking in existing data.world datasets

Creating your own datasets and linking them into the project

Adding data directly to the project (limit to data that is unique to that project and wouldn't make sense to reuse elsewhere)

To connect or add data, just click the Add data button from your project page or workspace and choose data.world dataset to browse for existing datasets, or New file to add a local project file:

When you pull an existing dataset into your project, you're really linking to its original location.This means that all updates to the original data will be populated to your project automatically. It also means that you can't modify the underlying data or metadata within the project interface, however you can use it in queries to create the desired output.

When adding new files, you can add data from your computer or cloud storage, urls, and integrations--just as we covered in adding files to datasets.

Projects have metadata the same as datasets do, so document them in the same way as a dataset. For projects, the summary can be used to document your project and/or findings, keep track of your questions, to-dos, and further document your sources. Here are a couple of examples to demonstrate how your summary can be utilized: Exploring THOR and How is the federal government fighting the opioid epidemic?

Querying and analyzing

Once you have curated your project files and resources, you can work with them in the project workspace. To get to the workspace, click the Launch workspace button on the upper right corner of your project page.

In the workspace you can manage your project files and data sources, modify the summary or data dictionary, view or download your project files, preview select file formats in-line, and write SQL or SPARQL queries against supported data files. Once you have output, either directly from a file or the results of a query, you can also connect them to external tools like Tableau, R, or Python through an integration.

All data files are normalized so they're immediately queryable and joinable, whether they're similar formats or not. This lets you jump right into analyzing and querying to perform calculations, produce summaries, and manipulate data across many different formats and locations.

data.world supports SQL and SPARQL query languages. SPARQL is the query language for graph data, which is how all of the data is stored behind the scenes on data.world. SQL is by far the most widely known query language, so we’ll use that in these examples. Check out our SQL and SPARQL tutorials to learn more on each.

To start writing a query select New query from the menu in the left column or when viewing a particular file, click the Query link at the top of the file display:

From there you'll be taken to a new tab in the workspace with a pre-populated sample query. Clicking the Run query button on the right will return the results of your query:

The sample query serves as a good starting point for you to use to build your own SQL queries. You can modify it to return a subset of the initial data or expand on it to start joining your other datasets and tables.

To reveal a list of the tables and columns in your project, select the left arrow button to the right of the Run query button. You can use this schema browser to quickly find, copy and paste column and table names into your query to avoid typos, as well as find quick stats on each field by clicking the 'i' next to it:

When ready, name your query and save it for later use:

When you save a query you are prompted to choose who can see the query, either anyone with access to the project or only you. Saved queries show up on the lower left of the workspace window and can be duplicated, edited, or removed by clicking on the three dots to the right of their names:

All the files you open in the workspace stay open in tabs until you close them so you can easily navigate between them.

Once you have the desired results from your query, you can use the Download button to download them in a CSV or XLSX file, or use the Open in app option to connect them to one of our many integrations such as Tableau, Power BI, Google Data Studio, or your Python or R environment. You can also use the copy URL or embed code option to send a direct download link to someone (data.world login not required) or even embed the query results in a discussion or use as the URL source to power another file within your project or dataset.

Note that all files, queries, datasets, and projects are their own API endpoints, so if you don't see an integration with your desired tool, you could still work with it via our REST API. Send us a message as well, as we're always working with partners to expand data.world integrations!

Visualizing data

When you're ready to visualize data from a table or a query, pull it into one of the many supported integrations, or utilize data.world's Chart Builder integration.

Chart Builderis an open-sourced integration which utilizes Vega Lite and provides a light-weight and easy way to create quick visualizations of your data.Embed your charts back into your project as files or insights, and easily share with other users.

Select Chart at the top of the data table or query results, or select Open in App and select Chart Builder:

This will open a new browser tab/window showing the chart builder interface. Once you have set your chart configuration, a preview will be displayed in the main window:

Your chart can now be downloaded as a graphic or JSON for embedding into other applications, or you can also embed it back into your project or dataset as a file or insight using the Save as option:

Sharing and collaboration

Similar to adding contributors to a dataset, you can add people to your project. Go to the People tab from your project page to add and manage individual and team access to your project:

Once others have access to your project, you can utilize discussions to engage team members, ask questions, and discuss elements such as charts, queries, or insights. All discussions support markdown, so you're able to embed images, charts, and even other datasets and queries.

Discussions can be had at the project and topic level under the Discussions tab:

Or on specific insights posted to a project:

Tag people in your posts to ensure they get notified, and embed visualizations or other embeddable content using the embed URL options ( see some examples ). For example, the following post embeds an interactive Tableau visualization, and also tags the user @hhaveliw for credit:

When posted, it would render as:

Governance and auditability

All data.world datasets and projects are logged and versioned so you can see what activity has happened across your team, as well as get back to previous revisions. Access past activity through the Activity tab on your dataset or project:

Select Versions to download previous versions:

Additional resources

There are many more resources where you can find data.world information and support. Here are some useful links, and please reach out if you're unable to find what you're looking for or have feedback for our team!

data.world tutorials & documentation

data.world Slack community

data.world language tutorials (SQL, SPARQL, Markdown)

data.world API documentation

data.world blogs: Meta (engineering) & (data news & marketing)

Recently data.world rolled out a new workflow in order to improve the user experience.The new workflow really highlights the differences between datasets and projects and clearly delineates when to use one and when to use the other. Previously the two could be used more or less indistinguishably--even though there were intended functional differences. However the ability to do all the same things in both places created confusion in users and led to inconsistent data practices. As a user's or organization's number of datasets and projects grew it rapidly became unwieldy trying to find the right data in them. The new workflow guides users to better--replicable and consistent--data management and analysis.

In this article we'll compare datasets to projects and show how the dataset and project workflows have changed covering:

What a dataset is

What a project is

What's new

Queries and the new workspace

When to use a dataset and when to use a project

What is a dataset?

To begin we need to clarify what a dataset is. Datasets are the building blocks for projects. They contain data and metadata related to a topic.The files and tabular data in a dataset are meant be used--queried and analyzed--in one or more projects. Datasets are meant to be reusable assets. They can be combined with other datasets in projects, or they can be a single source for querying and analysis in a project.

Datasets can be owned by an individual or an organization, and a dataset provides an additional layer of access permissions to the data in a project. Because permissions are assigned at both the dataset and the project level an individual can create a project available to the public, but if the individual adds any datasets owned by an organization to the project, they won't be visible to the public--only to the other people in the organization. Not only is the dataset not visible, but any queries in the project written against that dataset are also not visible except to members of the organization.

Because datasets are linked to projects, any changes to the data or the metadata in the dataset show up automatically in the linked project. Linking data to a project instead of copying it into the project means that everything is kept up to date throughout your organization.

What is a project?

Projects bring datasets together with documentation and analysis. This is where work and collaboration happen. A project, as the name implies, likely has a beginning and an end. Data in it is shared and analyzed, and insights are derived from the analysis and written up in the project.

The biggest difference between a dataset and a project is that datasets can be linked to and included in projects, but projects cannot be linked to or included in other projects or datasets--nor can the files that are added directly to a project. With a project you can run queries against the data, analyze it, share it and create charts and visualizations from it. However if you decide to start right away with a project and add your data files to it, neither you nor anyone else can link those data files to another project. The only way to reuse the data in another project is to download it file by file and re-upload it into a dataset or directly into another project. While there are times you'll want to download and re-upload files instead of just linking to them, you won't have a choice if you start by adding new data files directly to a project. One disadvantage to re-uploading is that you have to recreate all the metadata for the files (descriptions and the data.dictionary) which is a very cumbersome process!

What's new?

The main change in the layout and workflow of datasets and projects is that there are no longer separate dataset and project workspaces. Instead for datasets there is now an option to explore the dataset in a new, untitled project window. In addition to exploring the dataset you can also find out how many projects use it, link it directly to another project you have already created, or create a new project based upon it:

navigating the project workspace

If you select Explore this dataset there is functionally no difference between the old dataset workspace and the new untitled project workspace. The button takes you to a workspace where you are able to browse or query, but to begin analysis (i.e., save anything) you need to save the project/give it a name:

Queries and the new workspace

Obviously the biggest change here for users accustomed to the previous workflow is that queries are no longer saved to datasets by default. The logic behind this change is that datasets are for storing files and tables, and projects are for querying and analyzing those files and tables. A dataset is meant to be reused in multiple projects, andif queries are saved to it instead of to the projects using it then the dataset can rapidly fill with irrelevant queries making it difficult to use. However if the queries specific to a project are all stored in that project, the linked dataset remains clean and ready for reuse.

The reasoning above covers 80\% of the use cases, but what about the times you really do want to save a query to a dataset? Maybe you want to clean up the data, join tables and preserve the lineage of the original tables for reference, or just use the query in multiple projects without having to rewrite it (you might even want to parameterize it). In those cases it is useful to be able to save your query to the dataset, and you can still do that. After running your query, to save it to the dataset select the Save link and click the drop-down link to the right of the + New Project option. In addition to New project you'll also see the name of the dataset. Select it and the query will be saved to the dataset and you'll still be in an untitled, unsaved project:

One thing about saving queries to the dataset instead of to the project is that queries saved to a dataset won't show up in the queries list of any project the dataset is used in. Instead they'll be displayed under the connected datasets info:

For details on all the features in the updated project workspace see our quickstart to.

When to use a dataset and when to use a project

Generally if you are putting up data to share or data that is private but which you might conceivably want to reuse in other projects, it's better to add the data to a dataset. If the data is in a dataset, all of its metadata will automatically show up in your project because the dataset is linked instead of copied. All changes to the original dataset--including automatic updates from the source and manual updates by the dataset owner to the metadata--will also be conveyed.

The table below summarizes the differences between adding data files to a dataset vs. to a project:

Dataset vs. Project

dataset

project

Can run and save queries against

X

X

Can have charts/visualizations

X

Can incorporate different file types

X

X

Can contain multiple files

X

X

Can be shared/have contributors

X

X

Can have a discussion thread

X

X

Can include insights

X

Can use existing data.world datasets without having to download and reimport them and having to recreate the associated meta-data

X

Can be included in a project

X

Can be shared for others to use in their own datasets and projects

X

Your data.world settings page allows you to configure a number of options in the following categories:

Profile

Account

Organizations

Billing

Notifications

Advanced

If you're logged into your account, update these settings by clicking on your profile image (or placeholder image) in the top right corner of data.world and select Settings:

[email protected]

Profile

The entries in this section appear on your profile page and will be visible to other data.world users. Settings in this section include:

Full name (required)

Company or organization

Website

Bio

Photo - this can be uploaded from your computer, connected via Dropbox, linked via a URL, or taken with your computer's camera via the browser

Account

E-mail address. Only one account is allowed for each e-mail address registered on data.world. When changing your e-mail address, we will send you a verification e-mail to the new address. Follow the instructions in that e-mail to verify the new address.

Password

Data.world currently doesn't support changing your username, so if you need it updated or run into any other login issues, please submit a request through a ticket or by emailing .

Organizations

Create a new organization in which you will be an administrator

Leave any organizations that you are a part of

If you have the appropriate access level, manage the membership of the organization and modify the subscription level of the organization

Billing

Modify the subscription level of your individual account

Modify the subscription level of any organizations in which you are an administrator

Update credit card information for a subscription

Notifications

Toggle e-mail notifications for projects and datasets and projects that you're part of

Advanced

Access and reset account-wide API tokens

Revoke access to any authorized integrations

Enable experimental features

Size limits

Dataset ingested to data.world

1 GB

Metadata management datasets

no limit

Project

1 GB of project-specific files, no limit on linked datasets.

Derived dataset

1 GB

Virtual dataset (hosted on a remote server)

Size not limited by data.world

Size of a file in a dataset

1 GB

Number of files in a dataset

250

Number of columns in a table

limited by file size (1 GB)

Number of columns previewed in a table

50

Number of columns previewed in query results

500

Number of rows in a table

limited by file size (1 GB)

Number of rows previewed in query results

10,000

Rate limiting: Number of burst streams API calls

5 in the first second or after a 5 second idle period, then 1 per second

Size of a record streamed

1 MiB

Size of a request streamed

1 GB

Number of JSON objects in a stream

1 GB divided by the avg record size

Timeouts

Query timeout before first byte is transmitted (tier 1-2)

60 sec

Query timeout before first byte is transmitted (tier 3-4)

5 min

Query timeout before last byte is transmitted

60 min

Data upload timeout

None: As long as packets continue to be passed the connection will stay open

Data.world has developed an open source Python library for working with data.world datasets.

This library makes it easy for data.world users to pull and work with data stored on data.world. Additionally, the library provides convenient wrappers for data.world APIs, allowing users to create and update datasets, add and modify files, and possibly implement entire apps on top of data.world.

This guide covers the following topics:

Getting started

Reference

Getting started

Installation

You can install data.world Python library usingpipdirectly from PyPI:

pip install datadotworld

Optionally, you can install the library including pandas support:

pip install datadotworld[pandas]

If you usecondato manage your python distribution, you can install from the community-maintained [conda-forge]( https://conda-forge.github.io/ ) channel:

conda install -c conda-forge datadotworld-py

Configuration

This library requires a data.world API authentication token to work.

Your authentication token can be obtained on data.world once you enable Python under Integrations > Python

To configure the library, run the following command:

dw configure

Alternatively, tokens can be provided via theDW_AUTH_TOKENenvironment variable. On MacOS or Unix machines, run (replacing<YOUR_TOKEN>>below with the token obtained earlier):

export DW_AUTH_TOKEN=<YOUR_TOKEN>

Examples

Theload_dataset()function facilitates maintaining copies of datasets on the local filesystem. It will download a given dataset's datapackage and store it under~/.dw/cache. When used subsequently,load_dataset()will use the copy stored on disk and will work offline, unless it's called withforce_update=Trueorauto_update=True.force_update=Truewill overwrite your local copy unconditionally.auto_update=Truewill only overwrite your local copy if a newer version of the dataset is available on data.world.

Once loaded, a dataset (data and metadata) can be conveniently accessed via the object returned byload_dataset().

Start by importing thedatadotworldmodule:

import datadotworld as dw

Then, invoke theload_dataset()function, to download a dataset and work with it locally. For example:

intro_dataset = dw.load_dataset('jonloyens/an-intro-to-dataworld-dataset')

Dataset objects allow access to data via three different propertiesraw_data,tablesanddataframes. Each of these properties is a mapping (dict) whose values are of typebytes,listandpandas.DataFrame, respectively. Values are lazy loaded and cached once loaded. Their keys are the names of the files contained in the dataset.

For example:

>>> intro_dataset.dataframes

LazyLoadedDict({

'changelog': LazyLoadedValue(<pandas.DataFrame>),

'datadotworldbballstats': LazyLoadedValue(<pandas.DataFrame>),

'datadotworldbballteam': LazyLoadedValue(<pandas.DataFrame>)})

IMPORTANT: Not all files in a dataset are tabular, therefore some will be exposed viaraw_dataonly.

Tables are lists of rows, each represented by a mapping (dict) of column names to their respective values.

For example:

>>> stats_table = intro_dataset.tables['datadotworldbballstats']

>>> stats_table[0]

OrderedDict([('Name', 'Jon'),

('PointsPerGame', Decimal('20.4')),

('AssistsPerGame', Decimal('1.3'))])

You can also review the metadata associated with a file or the entire dataset, using thedescribefunction. For example:

>>> intro_dataset.describe()

{'homepage': 'https://data.world/jonloyens/an-intro-to-dataworld-dataset',

'name': 'jonloyens_an-intro-to-dataworld-dataset',

'resources': [{'format': 'csv',

'name': 'changelog',

'path': 'data/ChangeLog.csv'},

{'format': 'csv',

'name': 'datadotworldbballstats',

'path': 'data/DataDotWorldBBallStats.csv'},

{'format': 'csv',

'name': 'datadotworldbballteam',

'path': 'data/DataDotWorldBBallTeam.csv'}]}

>>> intro_dataset.describe('datadotworldbballstats')

{'format': 'csv',

'name': 'datadotworldbballstats',

'path': 'data/DataDotWorldBBallStats.csv',

'schema': {'fields': [{'name': 'Name', 'title': 'Name', 'type': 'string'},

{'name': 'PointsPerGame',

'title': 'PointsPerGame',

'type': 'number'},

{'name': 'AssistsPerGame',

'title': 'AssistsPerGame',

'type': 'number'}]}}

Reference

Standalone functions

load_dataset(dataset_key,force_update=False,auto_update=False)

Load a dataset from the local filesystem, downloading it from data.world first, if necessary.

This function returns an object of typeLocalDataset. The object allows access to metedata via it’sdescribe()method and to all the data via three propertiesraw_data,tablesanddataframes, all of which are mappings (dict-like structures).

Parameters

dataset_key(str) Dataset identifier, in the form of owner/id or of a url

force_update(bool) Flag, indicating if a new copy of the dataset should be downloaded replacing any previously downloaded copy (Default value = False)

auto_update(bool) Flag, indicating that dataset be updated to the latest version

Returns

The object representing the dataset

Return type

LocalDataset

Raises

RestApiError If a server error occurs

open_remote_file(dataset_key,file_name,mode='w',**kwargs)

Open a remote file object that can be used to write to or read from a file in a data.world dataset

Parameters

dataset_key(str) Dataset identifier, in the form of owner/id

file_name(str) The name of the file to open

mode(str,optional) the mode for the file - must be ‘w’, ‘wb’, ‘r’, or ‘rb’ - indicating read/write (‘r’/’w’) and optionally “binary” handling of the file data. (Default value = ‘w’)

chunk_size(int,optional) size of chunked bytes to return when reading streamed bytes in ‘rb’ mode

decode_unicode(bool,optional) whether to decode textual responses as unicode when returning streamed lines in ‘r’ mode

**kwargs

Examples

>>> import datadotworld as dw

>>>

>>> # write a text file

>>> with dw.open_remote_file('username/test-dataset',

... 'test.txt') as w:

... w.write("this is a test.")

>>>

>>> # write a jsonlines file

>>> import json

>>> with dw.open_remote_file('username/test-dataset',

... 'test.jsonl') as w:

... json.dump({'foo':42, 'bar':"A"}, w)

... w.write("\n")

... json.dump({'foo':13, 'bar':"B"}, w)

... w.write("\n")

>>>

>>> # write a csv file

>>> import csv

>>> with dw.open_remote_file('username/test-dataset',

... 'test.csv') as w:

... csvw = csv.DictWriter(w, fieldnames=['foo', 'bar'])

... csvw.writeheader()

... csvw.writerow({'foo':42, 'bar':"A"})

... csvw.writerow({'foo':13, 'bar':"B"})

>>>

>>> # write a pandas dataframe as a csv file

>>> import pandas as pd

>>> df = pd.DataFrame({'foo':[1,2,3,4],'bar':['a','b','c','d']})

>>> with dw.open_remote_file('username/test-dataset',

... 'dataframe.csv') as w:

... df.to_csv(w, index=False)

>>>

>>> # write a binary file

>>> with dw.open_remote_file('username/test-dataset',

>>> 'test.txt', mode='wb') as w:

... w.write(bytes([100,97,116,97,46,119,111,114,108,100]))

>>>

>>> # read a text file

>>> with dw.open_remote_file('username/test-dataset',

... 'test.txt', mode='r') as r:

... print(r.read())

>>>

>>> # read a csv file

>>> with dw.open_remote_file('username/test-dataset',

... 'test.csv', mode='r') as r:

... csvr = csv.DictReader(r)

... for row in csvr:

... print(row['column a'], row['column b'])

>>>

>>> # read a binary file

>>> with dw.open_remote_file('username/test-dataset',

... 'test', mode='rb') as r:

... bytes = r.read()

query(dataset_key,query,query_type='sql',parameters=None)

Query an existing dataset

Parameters

dataset_key(str) Dataset identifier, in the form of owner/id or of a url

query(str) SQL or SPARQL query

query_type({'sql','sparql'},optional) The type of the query. Must be either ‘sql’ or ‘sparql’. (Default value = “sql”)

parameters(query parameters,optional) parameters to the query - if SPARQL query, this should be a dict containing named parameters, if SQL query,then this should be a list containing positional parameters. Boolean values will be converted to xsd:boolean, Integer values to xsd:integer, and other Numeric values to xsd:decimal. Anything else is treated as a String literal (Default value = None)

Returns

Object containing the results of the query

Return type

QueryResults

Raises

RuntimeError If a server error occurs

API Client Methods

The following functions are all methods of the datadotworld.api_client() class

add_files_via_url(dataset_key,files={})

Add or update dataset files linked to source URLs

Parameters

dataset_key(str) Dataset identifier, in the form of owner/id

files(dict) Dict containing the name of files and metadata Uses file name as a dict containing File description, labels and source URLs to add or update (Default value = {})description and labels are optional.

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> url = 'http://www.acme.inc/example.csv'

>>> api_client = dw.api_client()

>>> api_client.add_files_via_url(

... 'username/test-dataset',

... {'example.csv': {

... 'url': url,

... 'labels': ['raw data'],

... 'description': 'file description'}})

add_linked_dataset(project_key,dataset_key)

Link project to an existing dataset

This method links a dataset to project

Parameters

project_key(str) Project identifier, in the form of owner/id

dataset_key Dataset identifier, in the form of owner/id

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> linked_dataset = api_client.add_linked_dataset(

... 'username/test-project',

... 'username/test-dataset')

append_records(dataset_key,stream_id,body)

Append records to a stream.

Parameters

dataset_key(str) Dataset identifier, in the form of owner/id

stream_id(str) Stream unique identifier.

body(obj) Object body

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> api_client.append_records('username/test-dataset','streamId',

... {'content':'content'})

create_dataset(owner_id,**kwargs)

Create a new dataset

Parameters

owner_id(str) Username of the owner of the new dataset

title(str) Dataset title (will be used to generate dataset id on creation)

description(str,optional) Dataset description

summary(str,optional) Dataset summary markdown

tags(list,optional) Dataset tags

license({'Public Domain','PDDL','CC-0','CC-BY','ODC-BY','CC-BY-SA','ODC-ODbL','CC BY-NC','CC BY-NC-SA','Other'}) Dataset license

visibility({'OPEN','PRIVATE'}) Dataset visibility

files(dict, optionalDescription and labels are optional) File name as dict, source URLs, description and labels() as properties

Returns

Newly created dataset key

Return type

str

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> url = 'http://www.acme.inc/example.csv'

>>> api_client.create_dataset(

... 'username', title='Test dataset', visibility='PRIVATE',

... license='Public Domain',

... files={'dataset.csv':{'url': url}})

create_insight(project_key,**kwargs)

Create a new insight

Parameters

project_key(str) Project identifier, in the form of

title(str) Insight title

description(str,optional) Insight description.

image_url(str) If image-based, the URL of the image

embed_url(str) If embed-based, the embeddable URL

source_link(str,optional) Permalink to source code or platform this insight was generated with. Allows others to replicate the steps originally used to produce the insight.

data_source_links(array) One or more permalinks to the data sources used to generate this insight. Allows others to access the data originally used to produce the insight.

Returns

Insight with message and uri object

Return type

object

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> api_client.create_insight(

... 'projectOwner/projectid', title='Test insight',

... image_url='url')

create_project(owner_id,**kwargs)

Create a new project

Parameters

owner_id(str) Username of the creator of a project.

title(str) Project title (will be used to generate project id on creation)

objective(str,optional) Short project objective.

summary(str,optional) Long-form project summary.

tags(list,optional) Project tags. Letters numbers and spaces

license({'Public Domain','PDDL','CC-0','CC-BY','ODC-BY','CC-BY-SA','ODC-ODbL','CC BY-NC','CC BY-NC-SA','Other'}) Project license

visibility({'OPEN','PRIVATE'}) Project visibility

files(dict, optionalDescription and labels are optional) File name as dict, source URLs, description and labels() as properties

linked_datasets(list of object,optional) Initial set of linked datasets.

Returns

Newly created project key

Return type

str

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> api_client.create_project(

... 'username', title='project testing',

... visibility='PRIVATE',

... linked_datasets=[{'owner': 'someuser',

... 'id': 'somedataset'}])

delete_dataset(dataset_key)

Deletes a dataset and all associated data

Parameters

dataset_key(str) Dataset identifier, in the form of owner/id

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> api_client.delete_dataset(

... 'username/dataset')

delete_files(dataset_key,names)

Delete dataset file(s)

Parameters

dataset_key(str) Dataset identifier, in the form of owner/id

names(list of str) The list of names for files to be deleted

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> api_client.delete_files(

... 'username/test-dataset', ['example.csv'])

delete_insight(project_key,insight_id)

Delete an existing insight.

Parameters

project_key(str) Project identifier, in the form of projectOwner/projectId

insight_id(str) Insight unique id

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> del_insight = api_client.delete_insight(

... 'username/project', 'insightid')

delete_project(project_key)

Deletes a project and all associated data

Parameters

project_key(str) Project identifier, in the form of owner/id

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> api_client.delete_project(

... 'username/test-project')

download_datapackage(dataset_key,dest_dir)

Download and unzip a dataset’s datapackage

Parameters

dataset_key(str) Dataset identifier, in the form of owner/id

dest_dir(strorpath) Directory under which datapackage should be saved

Returns

Location of the datapackage descriptor (datapackage.json) in the local filesystem

Return type

path

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> datapackage_descriptor = api_client.download_datapackage(

... 'jonloyens/an-intro-to-dataworld-dataset',

... '/tmp/test')

>>> datapackage_descriptor

'/tmp/test/datapackage.json'

download_dataset(dataset_key)

Return a .zip containing all files within the dataset as uploaded.

Parameters

dataset_key(str) Dataset identifier, in the form of owner/id

Returns

.zip file contain files within dataset

Return type

file object

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> api_client.download_dataset(

... 'username/test-dataset')

download_file(dataset_key,file)

Return a file within the dataset as uploaded.

Parameters

dataset_key(str) Dataset identifier, in the form of owner/id

file(str) File path to be returned

Returns

file in which the data was uploaded

Return type

file object

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> api_client.download_file('username/test-dataset',

... '/my/local/example.csv')

fetch_contributing_datasets(**kwargs)

Fetch datasets that the authenticated user has access to

Parameters

limit(str,optional) Maximum number of items to include in a page of results

next(str,optional) Token from previous result page (to be used when requesting a subsequent page)

sort(str,optional) Property name to sort

Returns

Authenticated user dataset

Return type

dict

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> user_dataset =

... api_client.fetch_contributing_datasets()

{'count': 0, 'records': [], 'next_page_token': None}

fetch_contributing_projects(**kwargs)

Fetch projects that the currently authenticated user has access to

Returns

Authenticated user projects

Return type

dict

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> user_projects =

... api_client.fetch_contributing_projects()

{'count': 0, 'records': [], 'next_page_token': None}

fetch_datasets(**kwargs)

Fetch authenticated user owned datasets

Parameters

limit(str,optional) Maximum number of items to include in a page of results

next(str,optional) Token from previous result page (to be used when requesting a subsequent page)

sort(str,optional) Property name to sort

Returns

Dataset definition, with all attributes

Return type

dict

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> user_owned_dataset = api_client.fetch_datasets()

fetch_liked_datasets(**kwargs)

Fetch datasets that authenticated user likes

Parameters

limit(str,optional) Maximum number of items to include in a page of results

next(str,optional) Token from previous result page (to be used when requesting a subsequent page)

sort(str,optional) Property name to sort

Returns

Dataset definition, with all attributes

Return type

dict

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> user_liked_dataset =

... api_client.fetch_liked_datasets()

fetch_liked_projects(**kwargs)

Fetch projects that the currently authenticated user likes

Returns

Authenticated user projects

Return type

dict

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> user_liked_projects =

... api_client.fetch_liked_projects()

fetch_projects(**kwargs)

Fetch projects that the currently authenticated user owns

Returns

Authenticated user projects

Return type

dict

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> user_projects =

... api_client.fetch_projects()

get_dataset(dataset_key)

Retrieve an existing dataset definition

This method retrieves metadata about an existing

Parameters

dataset_key(str) Dataset identifier, in the form of owner/id

Returns

Dataset definition, with all attributes

Return type

dict

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> intro_dataset = api_client.get_dataset(

... 'jonloyens/an-intro-to-dataworld-dataset')

>>> intro_dataset['title']

'An Intro to data.world Dataset'

get_insight(project_key,insight_id,**kwargs)

Retrieve an insight

Parameters

project_key(str) Project identifier, in the form of projectOwner/projectid

insight_id(str) Insight unique identifier.

Returns

Insight definition, with all attributes

Return type

object

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> insight = api_client.get_insight(

... 'jonloyens/'

... 'an-example-project-that-shows-what-to-put-in-data-world',

... 'c2538b0c-c200-474c-9631-5ff4f13026eb')

>>> insight['title']

'Coast Guard Lives Saved by Fiscal Year'

get_insights_for_project(project_key,**kwargs)

Get insights for a project.

Parameters

project_key(str) Project identifier, in the form of projectOwner/projectid

Returns

Insight results

Return type

object

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> insights = api_client.get_insights_for_project(

... 'jonloyens/'

... 'an-example-project-that-shows-what-to-put-in-data-world'

... )

get_project(project_key)

Retrieve an existing project

This method retrieves metadata about an existing project

Parameters

project_key(str) Project identifier, in the form of owner/id

Returns

Project definition, with all attributes

Return type

dict

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> intro_project = api_client.get_project(

... 'jonloyens/'

... 'an-example-project-that-shows-what-to-put-in-data-world'

... )

>>> intro_project['title']

'An Example Project that Shows What To Put in data.world'

get_user_data()

Retrieve data for authenticated user

Returns

User data, with all attributes

Return type

dict

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> user_data = api_client.get_user_data()

>>> user_data[display_name]

'Name User'

remove_linked_dataset(project_key,dataset_key)

Unlink dataset

This method unlinks a dataset from a project

Parameters

project_key(str) Project identifier, in the form of owner/id

dataset_key Dataset identifier, in the form of owner/id

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> api_client.remove_linked_dataset(

... 'username/test-project',

... 'username/test-dataset')

replace_dataset(dataset_key,**kwargs)

Replace an existing dataset

This method will completely overwrite an existing dataset.

Parameters

description(str,optional) Dataset description

summary(str,optional) Dataset summary markdown

tags(list,optional) Dataset tags

license({'Public Domain','PDDL','CC-0','CC-BY','ODC-BY','CC-BY-SA','ODC-ODbL','CC BY-NC','CC BY-NC-SA','Other'}) Dataset license

visibility({'OPEN','PRIVATE'}) Dataset visibility

files(dict,optional) File names and source URLs to add or update

dataset_key(str) Dataset identifier, in the form of owner/id

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> api_client.replace_dataset(

... 'username/test-dataset',

... visibility='PRIVATE', license='Public Domain',

... description='A better description')

replace_insight(project_key,insight_id,**kwargs)

Replace an insight.

Parameters

project_key(str) Projrct identifier, in the form of projectOwner/projectid

insight_id(str) Insight unique identifier.

title(str) Insight title

description(str,optional) Insight description.

image_url(str) If image-based, the URL of the image

embed_url(str) If embed-based, the embeddable URL

source_link(str,optional) Permalink to source code or platform this insight was generated with. Allows others to replicate the steps originally used to produce the insight.

data_source_links(array) One or more permalinks to the data sources used to generate this insight. Allows others to access the data originally used to produce the insight.

Returns

message object

Return type

object

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> api_client.replace_insight(

... 'projectOwner/projectid',

... '1230-9324-3424242442',

... embed_url='url',

... title='Test insight')

replace_project(project_key,**kwargs)

Replace an existing Project

Create a project with a given id or completely rewrite the project, including any previously added files or linked datasets, if one already exists with the given id.

Parameters

project_key(str) Username and unique identifier of the creator of a project in the form of owner/id.

title(str) Project title

objective(str,optional) Short project objective.

summary(str,optional) Long-form project summary.

tags(list,optional) Project tags. Letters numbers and spaces

license({'Public Domain','PDDL','CC-0','CC-BY','ODC-BY','CC-BY-SA','ODC-ODbL','CC BY-NC','CC BY-NC-SA','Other'}) Project license

visibility({'OPEN','PRIVATE'}) Project visibility

files(dict, optionalDescription and labels are optional) File name as dict, source URLs, description and labels() as properties

linked_datasets(list of object,optional) Initial set of linked datasets.

Returns

project object

Return type

object

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> api_client.replace_project(

... 'username/test-project',

... visibility='PRIVATE',

... objective='A better objective',

... title='Replace project')

sparql(dataset_key,query,desired_mimetype='application/sparql-results+json',**kwargs)

Executes SPARQL queries against a dataset via POST

Parameters

dataset_key(str) Dataset identifier, in the form of owner/id

query(str) SPARQL query

Returns

file object that can be used in file parsers and data handling modules.

Return type

file object

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> api_client.sparql_post('username/test-dataset',

... query)

sql(dataset_key,query,desired_mimetype='application/json',**kwargs)

Executes SQL queries against a dataset via POST

Parameters

dataset_key(str) Dataset identifier, in the form of owner/id

query(str) SQL query

include_table_schema(bool) Flags indicating to include table schema in the response

Returns

file object that can be used in file parsers and data handling modules.

Return type

file-like object

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> api_client.sql('username/test-dataset', 'query')

sync_files(dataset_key)

Trigger synchronization process to update all dataset files linked to source URLs.

Parameters

dataset_key(str) Dataset identifier, in the form of owner/id

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> api_client.sync_files('username/test-dataset')

update_dataset(dataset_key,**kwargs)

Update an existing dataset

Parameters

description(str,optional) Dataset description

summary(str,optional) Dataset summary markdown

tags(list,optional) Dataset tags

license({'Public Domain','PDDL','CC-0','CC-BY','ODC-BY','CC-BY-SA','ODC-ODbL','CC BY-NC','CC BY-NC-SA','Other'}) Dataset license

visibility({'OPEN','PRIVATE'},optional) Dataset visibility

files(dict,optional) File names and source URLs to add or update

dataset_key(str) Dataset identifier, in the form of owner/id

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> api_client.update_dataset(

... 'username/test-dataset',

... tags=['demo', 'datadotworld'])

update_insight(project_key,insight_id,**kwargs)

Update an insight.

Note that only elements included in the request will be updated. All omitted elements will remain untouched.

Parameters

project_key(str) Projrct identifier, in the form of projectOwner/projectid

insight_id(str) Insight unique identifier.

title(str) Insight title

description(str,optional) Insight description.

image_url(str) If image-based, the URL of the image

embed_url(str) If embed-based, the embeddable URL

source_link(str,optional) Permalink to source code or platform this insight was generated with. Allows others to replicate the steps originally used to produce the insight.

data_source_links(array) One or more permalinks to the data sources used to generate this insight. Allows others to access the data originally used to produce the insight.

Returns

message object

Return type

object

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> api_client.update_insight(

... 'username/test-project', 'insightid'

... title='demo atadotworld'})

update_project(project_key,**kwargs)

Update an existing project

Parameters

project_key(str) Username and unique identifier of the creator of a project in the form of owner/id.

title(str) Project title

objective(str,optional) Short project objective.

summary(str,optional) Long-form project summary.

tags(list,optional) Project tags. Letters numbers and spaces

license({'Public Domain','PDDL','CC-0','CC-BY','ODC-BY','CC-BY-SA','ODC-ODbL','CC BY-NC','CC BY-NC-SA','Other'}) Project license

visibility({'OPEN','PRIVATE'}) Project visibility

files(dict, optionalDescription and labels are optional) File name as dict, source URLs, description and labels() as properties

linked_datasets(list of object,optional) Initial set of linked datasets.

Returns

message object

Return type

object

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> api_client.update_project(

... 'username/test-project',

... tags=['demo', 'datadotworld'])

upload_file(dataset_key,name,file_metadata={},**kwargs)

Upload one file to a dataset

Parameters

dataset_key(str) Dataset identifier, in the form of owner/id

name(str) Name/path for files stored in the local filesystem

expand_archives Boolean value to indicate files should be expanded upon upload

files_metadata(dict optional) Dict containing the name of files and metadata Uses file name as a dict containing File description, labels and source URLs to add or update

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> api_client.upload_file(

... 'username/test-dataset',

... 'example.csv')

upload_files(dataset_key,files,files_metadata={},**kwargs)

Upload dataset files

Parameters

dataset_key(str) Dataset identifier, in the form of owner/id

files(list of str) The list of names/paths for files stored in the local filesystem

expand_archives Boolean value to indicate files should be expanded upon upload

files_metadata(dict optional) Dict containing the name of files and metadata Uses file name as a dict containing File description, labels and source URLs to add or update

Raises

RestApiException If a server error occurs

Examples

>>> import datadotworld as dw

>>> api_client = dw.api_client()

>>> api_client.upload_files(

... 'username/test-dataset',

... ['/my/local/example.csv'])

Forward to https://apidocs.data.world/

When you run a query in data.world, one of the things you can do with the results is to save them as a new table in a dataset or project. The main reasons you might want to save your query as a new data file are security and ease of analysis. Saving the query allows you to:

Present a subset of the data from the original file

Reorder the data

Eliminate the need for joins or subqueries in future queries

Work with columns as rows and vice versa without having to include either a PIVOT or an UNPIVOT clause in your queries

In this article we'll look at the various things you accomplish by saving results from a query as a new data file and provide examples.

Present a subset of the data

When data files are streamed into data.world they include all the columns in the original file. Sometimes there are columns which are irrelevant to your analysis or which contain substandard or incomplete data. Some columns might contain sensitive information that can't be shared outside of the original audience of the dataset but also have data that would be useful to a wider group. While you could manually clean up the data by downloading, editing, and re-uploading it, you would lose the ability for it to automatically update from the original source if that is how the initial data was set up in data.world. However you can both clean the data and maintain automatic updates by running a query against the original data file and saving the results to another dataset or project.

For example, the data file USDA Honey Census data in the dataset USDA bee colony inventories and loss is automatically brought in and synced daily through a URL. Because the original dataset contains information on much more than bees and honey, there are many columns in the table that contain either identical data for all the rows or no data at all.

A quick look at the data pulled in from the URL looks like this :

UNPIVOT

All columns displayed above except the second one contain duplicate data. Scrolling through the rest of the columns reveals eight more which have either no values or all the same value. To make this data more useful we can run the following query :

and save it to another dataset (we could also have saved it to the current dataset). To save the query selectSave to dataset or project from the Download link:

There you will be prompted to either select a dataset or project from your recently viewed items, type the name of a different dataset or project and the system will autofill items for you, or save the data as a new dataset:

When saving the query results as a table in a brand new dataset, you create what is called a derviative dataset.The new dataset updates from the original dataset which updates from the source, but it only pulls in the data you want to see:

To keep the file up-to-date you can either manually sync it (using the Sync now button on the right of the screen shown above), or by going to the dataset/project overview page and setting the Autosync on:

Note that you have the option upon saving to either allow the query underlying the table to update based on changes to the original table or not. If you allow the query to update then changes to the original table will be reflected in your results (e.g., column name changes, columns added, columns deleted, etc.).

Reorder the data

When you use the SQL SELECT * clause to return data from a dataset, the columns in the resulting table are presented in the order in which they are found in the original data source. If you would always like to see them in a different order, you can accomplish this by running a query to order them as you would like to see them and then saving the query as a table in the current dataset. This query would be similar to the one shown above for presenting a subset of the data and the steps to save it would be the same.

Eliminate the need for joins or subqueries

Sometimes all the data you need to analyze is not stored in one table. While you could still access it using a join or a subquery, it's easier and more convenient to be able to query one table. Another reason for saving the data into one table is that queries against large amounts of data run faster if there is no subquery. Finally, it makes it easier for the members of your team who are not well-versed in SQL to do their own analyses of the data.

For this example we'll use the dataset SQL CRM Example Data. With the way the dataset is structured we can see a lot of information about what’s in the pipeline for sales agents (in the table sales_pipeline ), but we can’t easily see the connection between their managers (information stored in the table sales-teams ) and the pipeline. To see which managershad the most closed deals and for which accounts, we could run a query like this:

And that's nice, but if someone else wanted different information the query would have to be modified and the person running the query would have to be comfortable enough with SQL to know how to do it. Far easier would be to to write another query with theWITHclause and to create a new table from it. Here is an example of such a query:

The results would look like this:

By saving the query as a new table it could be queried directly without having to use the join. The first query in this example would now look like this:

Work with columns as rows and vice versa

SQL has two clauses that allow you to rearrange the data in a table: PIVOT and UNPIVOT. PIVOT lets you see rows as columns, and UNPIVOT shows columns as rows. These are both complicated clauses and do not allow certain other clauses to follow them (e.g., WHERE, GROUP BY, and HAVING). if you find yourself using PIVOT or UNPIVOT frequently, or if you'd like to be able to use one of the incompatible clauses, it would be far better to write the query using PIVOT or UNPIVOT once and then save the resulting table for further analysis. See our SQL documentation on PIVOT and for more information.

Tags are a powerful feature that you can use in a variety of ways to facilitate access to your data. For example, tags can be used to organize and group your dataset or project by topic, category, source, department, or team. They can be searched for explicitly with the tag search operator, and can also help to filter down more generic search results.

You can add or remove tags from a dataset or project's Overview page with either the Edit orAdd tags links on the right side: building your data dictionary

There is no limit to the number of tags you can use for a dataset, and there is an autofill feature on the tag field. If the dataset is owned by an organization, the tags displayed for autofill are chosen from all the tags used by the organization. If the dataset is not owned by an organization, the autofill suggestions are from a generic list of tags as well as from tags you have recently created.

For additional documentation elements, visit Documenting your data or continue to more details on .

We offer a secure, direct integration with Google Drive that will allow you to connect files to data.world without leaving our platform. For more information on that, see the Google Drive integration page.

If you would prefer not setting up that integration, you can add files one at a time by generating a public URL for each file.

You can find that URL by doing the following:

With the file open within Google Drive, click File > Publish to the web. adding files via URL

Select the file and format from the dropdown menus under Link.

Click the Publish button.

Copy the resulting link. This is the URL to use whento your data.world dataset.

A sharable download link, or pre-signed URL, is necessary for data.world to pull in files from external sources. This is the URL that will trigger the file download for anyone with the link.

Depending on the source, these can sometimes be tricky to find, so we'll do our best to keep these how-to's accurate. Please let us know if you see something missing, or have other sources you'd like us to add!

GitHub: how to find the sharable download URL for files on GitHub

Google: how to find the sharable download URL for files on Google

S3: how to find the sharable download URL for files on S3

Below is a list of status messages you might encounter when uploading data files to data.world.

Please open a support ticket for additional assistance.

Error message

More details

No data could be extracted from this file**

This status will display if a file type is supported by data.world, yet cannot be previewed.

Check for syntax or formatting errors within your file.

Want to see data previews? Reupload this file with an extension.

Currently, data.world depends on file extensions to determine how best to prepare your data. If a file is uploaded with no extension, then you will see this status message.

If you believe this file’s data is actually a known format (say, .csv), then re-upload this file with the new extension added.

Excel files >100MB may only be downloaded.**

Due to how Excel files are structured, in some cases we are not able to fully preview the data inside the file. It is, however, still available for sharing and download.

This file type >100MB can only be downloaded.

The file is too large to properly ingest into data.world and is unavailable for queries or previews.

This file is shareable, though some advanced features may be unavailable due to its size.**

This status indicates that a file contains more cells of data than data.world was expecting. In some cases, you might be able to remove any unnecessary blank columns, rows or tabs.

Only the first 50 of 111 files were extracted.

When uploading archived or compressed files (zip, tar, etc), ensure each contains 50 files or less. Any files over this limit will not be extracted.

2 files were too large to be extracted from this archive.**

If a file within an archive exceeds data.world's data limits, we will show this status.

Try splitting the file into multiple smaller files within our size limits, then reupload.

Sorry, we can't extract the contents of this archive. It may be corrupted.

The archive cannot be extracted for another reason - it may be an invalid archive or an unsupported file type

No data could be extracted from this file.

The file is of a supported type, but has a structural problem that prevents its from being extracted.

This file is shareable, though some advanced features may be unavailable due to the size of this dataset. **

If a data file is uploaded to a dataset that results in the total dataset exceeding what data.world can process, this status will be displayed.

Check for and remove any unnecessary blank columns, rows or tabs from all tabular files within the dataset, or contact support for further assistance.

**Note that these errors are related to enhancing tabular and graph data to provide advanced functionality (data previews and queries). The file will still be uploaded to data.world and be available for download.

Platform:

New features:

We're out of Preview (beta)!: As part of that, we've introduced free and paid tiers for both individuals and teams. See what each account type includes and pricing at https://data.world/pricing.

HTML Previews: We've added the ability to browse and preview HTML files. This means you can now add and preview much requested D3 visualizations and R markdown files amongst others.

Updates & fixes:

Email notification unsubscribe link: We've added unsubscribe links to all transactional emails. This allows you to stop activity notifications (comments, new files, etc.) at the dataset level.

Query editor enhancements:We're now bringing more context for columns into the query editor via small 'i' buttons on the dataset/project schema:

Streams API

dwSQL enhancements: We now support the TABLESAMPLE command which will randomly sample the returned set of rows instead of just limiting to the first N-returned rows. Here's an example.

Improved data table browser: We've improved the file-viewer for better loading times as well as new usability features including row numbers, column/row selection, improved headers and - most importantly - copy/paste ability, even directly into Microsoft Excel!

NOTE: This is currently behind our experimental flag and will be available to all members in the upcoming weeks. If you'd like to try it out now, go to 'Settings' -> 'Advanced' and check 'Enable experimental features'.

Integrations & API:

New features:

KNIME Connector :KNIME is a data workflow tool and allows you to more easily integrate data.world with various enterprise data sources.

Excel Integration : Easily push your Excel data out to data.world. This is still in beta, so try it out, send feedback,and leave a review in the Microsoft app store if you love it!

: Add data to a dataset via the new stream api. Allows you to append records to the file without needing to upload the entire file.

Updates & fixes:

Nothing new this week.

Platform:

New features:

New Project Insights creation experience:We wanted to give Project insights a major upgrade for the holidays! We established three distinct types of insights: Image-based insights, Embedded insights from third party platforms like Tableau & Plot.ly, and free-form Markdown-based insights. We’ve completely revamped the insight flow to guide people through creating these new insights, so check it out and let us know what you think!

contact us

If we choose the Embed option, we're presented with a field for the source link to embed. Here we're embedding a link to a Tableau Public visualization:

Click Add takes us through adding the Insight title, optional comment, and a preview of the embed:

Updates & fixes:

Nothing new this week.

-------------------------------------------------------------------------------------------------------------------------

Integrations & API:

New features:

Webhooks for datasets & Data Projects: We've added a UI to manage Webhook integrations (both Slack and general webhooks) making it easy to configure an external service to respond or take action when data.world datasets and projects are updated. This is a preview release, so please if you’re using it and let us know what you think!

Updates & fixes:

Nothing new this week.

Would you like to help up in testing new functionality as it’s being built? We have 30-minute remote sessions every Friday if you’d like to help in building data.world!

Schedule a session with us and we’ll also throw in a data.world t-shirt and $20 Amazon gift card!

Platform:

New features:

Nothing new this week.

Updates & fixes:

We now warn you if you try to navigate away from your query without saving!

Markdown editor with image and URL embeds expanded to discussions.

Backend improvements for organization and team management (no frontend impact).

-------------------------------------------------------------------------------------------------------------------------

Integrations & API:

New features:

Advanced sync added into data.world API.

V1.6 of Python SDK released, adds projects and insights support.

Redshift integration (beta) improvements and added to the integrations gallery.

Plot.ly added to download menu when viewing a file or after running a query:

streams API

New VegaLite based exploratory visualization tool is in alpha! Check out the teaser video and we’re actively looking for feedback if you could take a moment to share feedback in our Slack community (join at slack.data.world ) or by emailing [email protected].

Updates & fixes:

For users of the, we now displayhow many records are pending ingest via the stream within the data.world application.