Socrata FAQs

Citizen Connect is one of the latest products in the Socrata suite, and since its launch in March a lot has been happening! Below youll find a few of the top new features added in the last couple of months, as well as information about a webinar with the City of Santa Rosa who will be sharing what they learned about their homeless emergency through the use of Citizen Connect.

Webinar: Tackling Homelessness with Data

[email protected]

Join Eric McHenry, CIO at City of Santa Rosa, and Clare Hegg, Product Marketing Manager at Socrata, as they talk about how the City of Santa Rosa tackled their homeless emergency through the use of data. Youll hear not only about the challenges Santa Rosa faced in rethinking its homeless strategy, but also how they implemented successful changes by using Citizen Connect to showcase program-specific data.

This webinar will be on Tuesday, May 23, 2017 at 11am PDT / 2pm EDT, and you can register here. We hope to see you there!

The Newest Features for Citizen Connect

Searching Descriptions

To help users who are looking to answer specific questions, in addition to searching by the place (or address), pin number, or the region, you can now allow your users to search by description. By enabling this type of search, users can quickly filter down the map to things that are important to them, such as all open 311 ticket requests, a crime type, or even all permits that are related to flood work.

Advanced Trends

Citizen Connect now automatically provides an advanced trends view into the Tickets datasets, breaking down the data by status (such as Open, In Progress, or Closed for a 311 ticket, Attempted or Completed for a police incident, for example) and providing a view into when services are requested from the community by time of day and day of week.

More Configurability

There are a number of new administrative features, allowing you to present your data in a way that makes sense given the context of your city, county, state, or other jurisdiction. As an Admin, you can now:

determine if you want the right hand sidebar on the homepage, or the overview, to default to open or closed when someone first comes to your site;

set the term used for pin to something like ticket or incident and that term will then be used throughout the site experience for your users;

set the name of the datasets listed on the download page, making it easier for the user to find what they need;

and set the default radius for the near me search to best fits the geographic size of your jurisdiction, as well as the density of the pins on the map.

For more information about these latest product updates, check out the full release notes.

Check out our other Product News, and give us a shout at with questions or ideas.

Highlights

Administrators will have the ability to configure custom colors and palettes to be used by end-users on a domain-wide basis for creating visualizations.

Major Updates

The filter autocomplete dropdown in the Visualization Canvas will now show all available values, instead of the top 20. Chart creators can also choose to sort autocomplete values alphabetically as well as by frequency.

Users can now create, style and embed calendars in the visualization canvas to share event schedules, legislative calendars and more.

Minor Updates

Administration & User Management

Resolved an issue where app token links on the Site Analytics page were broken.

Resolved an issue where user accounts creation was case-sensitive. User invitations and account creation are now case-insensitive.

Resolved an issue causing the search bar on the connectors page not to work.

Resolved an issue with Socrata Data Player templates metadata panel.

Catalog & Metadata

Added notifications when catalog results exceed the 100,000 items.

Resolved an issue causing catalog pagination above 1000 not to work.

Resolved critical accessibility issues on the catalog page.

Dataset Management & Updates

Resolved an issue with the display of phone numbers in single row view of Grid View.

Resolved an issue where XML exports occasionally failed.

Resolved an issue in which Google search was returning widget results for data pages.

Resolved an issue in which derived views created through a search query on the dataset did not include the search term in the UI.

Stories

Resolved an issue in which table of content links navigated to the story draft.

Resolved an issue, specific to Firefox, in which classic visualizations previews were not loading in Stories.

Visualizations

Resolved an issue where users with the Editor role didn't see the asset action bar on assets they did not create.

Resolved an issue where drilldowns did not display data properly when drilling into a number column.

Resolved small bugs and edge cases for new Calendars.

Coming Soon!

Expected Release - 2 weeks

Users will soon have more advanced alert scheduling and threshold triggering options when creating an alert for a dataset.***

Expected release - 1 month

Updated Asset Inventory datasets will be added to all domains with schema changes adding Approvals status, visibility status, and row and column count for each dataset. Existing Asset Inventory datasets will continue to be updated until February 29th, 2020.

Legend: *** client-requested functionality

Curious about other areas of the platform? Visit our Performance or SCGC release notes for more! Or give us a shout at [email protected] with questions or ideas.

Before reaching out to us about an issue you've run into with the Socrata platform, take a moment to review the Known Issues indicated in the table below. The issue youd like to report just might be something were already aware of and are working hard to resolve!

Known issues are previously reported issues that may result in difficulty using some aspects of the platform. While it is not a comprehensive list, these are issues we suspect might have the potential to be highly impactful. As possible, well indicate a workaround below.

If you have questions, need help, or if you'd like to be notified when the issue is fully resolved, please reach out to us! Just email us at [email protected] and we'll get back to you soon!

Issue

Status

Workaround

Region map (choropleth) cards on a Data Lens and region maps in the visualization canvas built for a dataset with multiple location columns can show data aggregated to the wrong location column without warning.

Identified/Investigating

Refrain from creating region maps for datasets with multiple location columns. Users can create multiple datasets, each with one location column, in order to create the desired maps.

There is work planned to support multiple location columns and region maps for the future (timeline TBD), but the configuration is currently unsupported.

DataSync failures for "Missing Fields" where georeference columns - ":@computed_region_..." are not mapped.

N/A

Please ensure you are using the latest version of DataSync (DataSync releases can be found here.)

Resolved Issues

Issue

Date Resolved

Error when attempting to transfer ownership of community assets to roled users.

This issue was resolved on November 5, 2019.

Measures calculation values are not rounding up. Example 10.35 is showing up at 10.3 instead of 10.4

This issue was resolved on December 10, 2019.

Social Data Player (SDP) templates are not saving configured changes.

This issue was resolved on December 10, 2019.

Let us know how we can help! Email us at [email protected]

Administrators now have the ability to configure domain-wide custom colors and palettes that can be used by users when creating visualizations. To access this functionality, navigate to the Admin panel and click on Site Appearance > Colors tab

From here, site admins. can create a new custom palette as follows:

Click on Add New Palette

Name the new palette to be created

Enter the color names and/ or Hex codes under the Palette Hex Codes section. The colors can be viewed on the Palette Preview section as they are being entered.

Click on Update to save the changes.

To Delete a custom palette, click on the palette form the Color palette list, then click on Delete.

New custom palettes can be found on the new visualization canvas by clicking on Presentation > Color (dropdown) > Custom as shown below

Once you have found an interesting dataset, chart, or map to look at, what can you do on the page?

contact the dataset owner

For reference, "View" means any filter, chart, map, etc that is "Saved" from a dataset.

1. Title: This is the title given to the dataset or view. Views, such as charts, can have a different name from the dataset it is based on.

2. Based On: If you are on a view, this will link to the dataset that the view is based on. For example, if you create a pie chart of the total number of governors for each party, this would link back to the dataset with the numbers for each year.

3. Description: This is thedescriptiongiven to the dataset or view.

4. Social Media: This allows you to interact with this dataset or view through social media:

RSSSubscription: Subscribe to updates on the dataset

Facebook: Share the dataset or view on Facebook

Twitter: Share the dataset or view on Twitter

Email: Share the dataset or view over email

5. Dataset View: These buttons allow you to switch between the different views of a dataset or view, you can have more than one view splitting the screen vertically at a time

Detail Row: This view displays each column and row in a tabular format (seen in example above)

Fat Row: This view displays the details for each row grouped together, for example columns may be in a vertical list.

Single Row: This view displays all the details for one row at a time, There are arrows to scroll through next and previous rows.

Chart: If you are viewing a chart a button for the chart view will appear.

Map: If you are viewing a map, a button for the map view will appear.

6. Search: Enter in a word or words to search within the dataset for matches.

7. Sidebar Buttons: Each of these buttons opens up a sidebar on the right-hand side of the dataset or view. Not all will appear, depending on if you are logged in and what role you have on the site. For more about user roles, read this article.

Edit: Click this sidebar to create a working copy and edit the dataset. From a working copy, you can edit individual cells, append new rows, or replace the entire dataset.

Manage: Click this sidebar to transfer ownership of the dataset or view to another user, delete the dataset or view, share the dataset or view with other users, make the dataset public or private, show and hide columns, and change the column order.

More Views: Click this sidebar to see the other views created from the same dataset. These might be filtered views, charts, or maps. The number in red reflects the total number of views created from the dataset.

Filter: Click this sidebar to sort the dataset by columns, group and roll up the dataset, filter the dataset, and set the default filter for that view.

Visualize: Click this sidebar to set conditional formatting, create a calendar, create a map (if you have location data), and create a chart.

Export: Click this sidebar to access APi information, print the dataset, and export the dataset.

Discuss: If commenting is enabled, click this sidebar to comment and read other's comments on the dataset. Commenting can beenabledfor the dataset and cell level.

Embed: Click this sidebar to create a form from the dataset and create Social Data Player embeds

About: Click this sidebar to view the metadata information about the dataset, edit the metadata, view dataset analytics, and .

A variety of plugins are available for Socrata Gateways. This list will change as plugins are added.

List of Plugins

A -B - C - D - E - F - G - H - I - J - K - L - M - N - O - P - Q - R - S - T - U - V - W - X - Y - Z

- A -

Amazon S3:The Amazon S3 plugin provides access to data files hosted on S3. Required Fields: Bucket, Region.

- B -

- C -

CKAN:The CKAN plugin reads items from a CKAN instance and ports them over. Allowable file types: csv, xls, xlsx, tsv, kml, kmz, json, geojson, zip. Required Fields: CKAN URL.

CSV:The CSV plugin reads a CSV file from a specified directory and uploads the contents of the CSV file.Required Fields: Root Directory

- D -

Datasync:The DataSync plugin reads a SIJ file from a specified directory and uploads the contents of the specified CSV file.Required Fields: Root Directory

- E -

Elastic Search:The Elastic Search plugin supports access to Elastic Search endpoints to pull json given defined indices.Required Fields: Port, Server

Esri:The ESRI plugin provides access to ESRI Map and Feature Servers via their REST endpoints. For more information about the ESRI REST service, please see the developer documentation.Required Fields:ESRI REST URL

Excel:The Microsoft Excel plugin reads the contents of a specified sheet within an Excel spreadsheet and uploads its contents.Required Fields:Root Directory.

Exchange:The Microsoft Exchange plugin supports API access to metadata pertaining to email and calendar items linked to a specified account.Required Fields:Email Address, EWS Server, Password

- F -

FBI:The FBI API plugin provides access to the API maintained by the FBI containing aggregate crime statistics reported through UCR annual reports. For more information about the API and getting an API key, please see: https://crime-data-explorer.fr.cloud.gov/api.Required Fields:API Key

FTP:The FTP connects to a provided FTP server and allows access of files contained on the server.Required Fields:Binary transfers, Host, Use passive mode, Password, Port, Username

- G -

Google Sheets:The Google Sheets plugin supports reading directly from a particular sheet within the document and uploading the contents. The sheet must be set so that anyone with a link can view the sheet. For more information on creating a Google API key, please see https://cloud.google.com/docs/authentication/api-keys. For information on the Google Sheets ID, please see https://developers.google.com/sheets/api/guides/concepts#spreadsheet_id.Required Fields: API Key, Spreadsheet ID

- H -

- I -

Imap:IMAP plugin for reading from email servers.Required Fields:Password, Port, Server, Username

- J -

Jira:The JIRA plugin reads ticket metadata through the JIRA API. The default base fields that will be pulled are: creator, reporter, assignee, priority, status, updated, created, duedate, resolutiondate, summary, description. Custom fields can also be specified in the plugin configuration.Required Fields:Custom Fields (optional), Custom JQL Query (optional), Password, URL, Username.

- K -

- L -

- M -

Marketo:The Marketo plugin supports access to Dynamics user reports and queries.Required Fields:Client ID, Client Secret, Custom FIelds, URL

Microsoft Access:The Microsoft Access plugin provides a connection to tables found in Access files.Required Fields: Root Directory

Microsoft SQL Server:The Microsoft SQL Server plugin provides access to table and views within the SQL Server.Required Fields:Database, Address, Password, Username

MySql:The MySQL plugin provides access to table and views within the MySQL database specified.Required Fields:Database, Address, Password, Username

- N -

- O -

Oracle:The Oracle plugin provides table and view level data sources to be used to power Socrata datasets. The Oracle JDBC must be downloaded and the terms and conditions accepted. More information can be found at: https://www.oracle.com/technetwork/database/application-development/jdbc/downloads/index.html.Required Fields:Database, Address, Password, Username

- P -

PostgreSQL:The PostgreSQL plugin provides table and view level access to postgres databases.Required Fields:Database, Address, Password, Username

PostgreSQL Queries: The PostgreSQL plugin provides query level access to postgres databases.Required Fields:Database, Address, Password, Username

- Q -

- R -

- S -

Salesforce:The Salesforce plugin reads from the Salesforce API. Due to Salesforce restrictions, queries with two or more custom fields with long text fields are limited to 200 responses per request.Required Fields:Password, Security Token, Username

SAP HANA:SAP HANA database connector.Required Fields: Tables and Views, Database (default: SystemDB), Address, Instance Number (optional), Password, Queries, Username

SeeClickFix:The SeeClickFix plugin provides API access to the SeeClickFix Open311 compliant API based on the V2 schema. For more information about the API and rate limits please see: http://dev.seeclickfix.com/.Required Fields:Open 311 Account ID

ServiceNow:Access to Service Now.Required Fields:Password, URL, Username

SFTP:The SFTP plugin connects to a provided SFTP server and allows access of files contained on the server.Required Fields: Host, Password, Port, Username

Shapefile:The shapefile plugin grabs a local zipped shapefile and ingresses it into Socrata.Required Fields: Root Directory

SQLite:The SQLite plugin connects to a provided SQLite file and provides access to tables.Required Fields: Root Directory

Sumo Logic:The Sumo Logic plugin allows searching.Required Fields:Access ID, Access Key

- T -

Tyler Incode:The Tyler Incode 10/Infinite Visions plugin reads from the database OData API for supporting financial transparency applications.Required Fields:Client ID, Client Secret, Integration, Password, Start Year, URL, Username

Tyler Munis:The Tyler Munis plugin enables the integration with the Tyler Munis ERP product for financial transparency applications.Required Fields:Database, Address, Integration, Password, Query, Username

Tyler New World ERP:The Tyler New World ERP plugin enables financial application integration with New World ERP system.Required Fields:Database, Address, Integration, Password, Query, Username

Tyler New World Public Safety:The Tyler New World Public Safety plugin provides citizen connect and CompStat integrations with the New World system.Required Fields:Database, Address, Integration, Password, Query, Username

Tyler Odyssey:The Tyler Odyssey plugin provides Socrata integrations with the Odyssey court system.Required Fields:Database, Address, Password, Schema, Username

- U -

US Census:The US Census plugin provides access to the Census API for various geographic and demographic details. More information about the API and the API Key please visit the User Guide: https://www.census.gov/data/developers/guidance/api-user-guide.html Required Fields: API Key

US Energy Information Administration:The US Energy Information Administration plugin provides access to the US Energy Information Administration API. More information about the API and the API Key please visit the User Guide: https://www.eia.gov/opendata/register.php.Required Fields:API Key

- V -

- W -

- X -

- Y -

- Z -

Socrata Gateway is an easy-to-use on-platform solution that allows you to publish, update, and automate datasets directly from key on-premise and cloud-hosted source systems.

Gateway utilizes a user agent,a thin Java client that lives behind the users firewall that connects to Socrata backend systems. This user agent is set up on a server that has access to the source system hosting your data. Various plugins exist for your agent to connect to different source systems both on the cloud (Esri, AWS S3, etc) as well as on-premise (MS SQL, Excel, etc).

Gateway updates your dataset using the Socrata Dataset Management Experience, so you can still use all of the same tools to edit, manage, transform and schedule your data.

How do I setup Socrata Gateway?

There are two guides for setting up Socrata Gateway

Socrata Gateway for Administrators

Socrata Gateway for Data Publishers

You can also find the following 60-second videos on setting up a Gateway Agent:

How to Provision a Socrata Gateway Agent on a PC

How to Provision a Socrata Gateway Agent on a Mac

Gateway Quick Start Guide

Users can download the Gateway Quick Start guide attached to this article with the following download link.

here

Who can use Socrata Gateway?

Once Gateway is enabled on your domain, site administrators will have full access to manage Gateways, including access to the admin page. Additionally, any user that has access to the "Choose Data Source" page in the Socrata Dataset Management Experience will be able to set-up Gateway. However, these users will not yet have access to the admin page.

Can I use Gateway to update an existing dataset?

Yes! Just open a draft of your dataset and select Add Data > Replace > Connect to an External Data Source (Socrata Gateway). Please make sure the schema of your existing dataset matches the datasource you want to connect to.

What plugins are available for Socrata Gateway?

You can view a list of all available plugins .

Did you know that you can upload non-parsable files such as images through the same tool you use to upload datasets? Here are the steps:

Click the "Create" button in the Internal Navigation Bar and select "Dataset": Primer Page

Provide a name for the file you'll be uploading, then select "Create Dataset"

On the next page, click on "Add Data":

Either select "Browse" and choose the image file you'd like to upload, or drag-and-drop the image file into the browser window to initiate the upload.

Once the file has uploaded, select "Save" in the bottom right:

Then select the "Publish Dataset" button in the action bar and follow the prompts to publish your image file:

And that's it!

Note: If you need a direct link to the image file itself (as opposed to the of the asset), navigate to the Primer Page of the image asset and right-click on the "Download" button. Then, select "Copy Link" (different browsers use different phrases, such as "Copy Link Location"), and that is the direct link to the image file hosted on Socrata. You can confirm that it works by pasting it into a new tab in your browser and loading it.

Visualization drill downs allow users to create hierarchies in visualizations so that users can click on a component of a chart and "drill in" to explore the data within that piece of the visualization. Visualization authors can now configure these hierarchies in the visualization authoring experience by adding several dimensions in an ordered hierarchy. Additionally, new controls have been added to the chart itself to drill back up through the data, to toggle between different dimensions in the hierarchy, and to reset the applied drill down filters to return to the original state of the visualization. These hierarchies are flexible, allowing users to drill down into any of the dimensions in the hierarchy in any order of their choosing. This is a powerful feature that allows users to easily explore data on the fly and gain rapid insights when consuming bar, column, and pie charts.

Configuring Drill Downs in a Visualization

Configuring drill downs in a visualization is simple. First, start by creating a visualization:

Keep in mind that the drill down feature is only available for the following types of visualizations:

Bar Chart

Column Chart

Pie Chart

Select the Dimension of your chart, which will serve as the top level of the drill down hierarchy. We're using a dataset of Building Permits in this example and have selected "Zip Code" as the Dimension:

Next, select the "Add Hierarchy" button to add lower levels of the hierarchy.The total number of hierarchy levels that can be configured is limited to four, including the Dimension (i.e., Dimension + 3 additional hierarchy levels). In this example, we'll add two additional levels"Permit Class" and "Permit Type Description":

Notice that there are new controls in the top left of the chart now that we have added levels of hierarchy:

Drill Up: This button takes you back up one level of the hierarchy. It is only enabled if you have drilled in to the chart data, as you cannot drill up past the top level of the hierarchy.

Hierarchy: This button displays each level of the hierarchy as it has been configured. You can also click on each level of the hierarchy, and it will set that as the x-axis without filtering the upper levels of the hierarchy unless you have already drilled in to the chart. If you have already drilled in to the chart, then switching to different levels of the hierarchy using this button will simply change the x-axis view while retaining the data filter configured by drilling into the chart.

Reset: This resets the chart to the original saved configuration.

Once you're finished configuring the visualization, save the draft. You can then test out the drill down behavior in the visualization authoring experience before publishing it:

In this example, selecting the "98103" Zip Code at the top level of the hierarchy drills in to all of the Permit Classes associated with that specific zip code. Then after selecting the "Single Family/Duplex" Permit Class value, it drills in to the third level of the hierarchy"Permit Type Description"revealing the total number of "Single Family/Duplex" permits for the "98103" Zip Code categorized by the Permit Type.

The Dataset Table

Were excited to announce that were rolling out some really impactful improvements to the data table. Based on extensive feedback from users, we built the new version to include:

Faster page loads, especially for large datasets

A modern interface that looks and behaves like the other new surfaces on the platform

More intuitive editing interactions

here

Updated Look

We have updated the styling on the menu panels (Edit, Manage, More Views, etc.) so that they match our other recently-built surfaces

In the legacy dataset table, two column types Photo and Document are displayed as thumbnails. In the new table, these are displayed as text (the name of the file) with embedded hyperlinks that open the photo in a new tab or download the file, respectively.

The social data player (SDP), Report Builder pages and legacy Performance goal pages will start rendering the new table in place of the existing legacy table.

Updated Behavior

The table will paginate rather than scroll, just like on Primer and the Visualization Canvas. The row count will default to 50.

You can now copy and paste values from cells from the published version of a dataset.

Rendering number columns as Percent will continue to work, but the green bar representing the percent will go away.

The following features related to editing Derived Views only. Datasets will be edited through the Socrata Dataset Management Experience.

When multiple sorts are applied to the table, only the top-level sort column will have a sort indicator (arrow) in the header. Previously, all columns with a sort applied contained a sort indicator in the header.

The default sort on the dataset can now only be updated and saved while in a draft of the dataset or derived view.

Editing column properties will happen via a new Column Editor modal overlaying the dataset table accessible in the Edit panel. Users will be able to edit all columns at once instead of one at a time.

Rearranging and hiding columns will now happen in the same column editing modal.

Deprecated Functionality

The ability to drill into a group in a rolled-up view of a dataset will not be available in the new data table.

Full-screen mode is not available in the new data table.

Column Totals.

Filtering directly from the column header will no longer be available in the data table. All filtering on the dataset will be controlled using the blue Filter pane.

A few column formatting options will no longer be available to be created in existing datasets, or during ingress of new datasets:

Formatted text (HTML)

Phone icon plus number

Star

Flag

Multiple Choice

We will preserve the contents of these five column types as plain text or numeric data. Here are the specific changes for each type:

Formatted text (HTML): well turn off this option in the ingress workflow and in the data table and convert existing cells to Plain Text in the new table.

Star: well turn off this option in the ingress workflow and in the data table and convert existing cells to the corresponding numeric value in the new table. For example, a five-star column will be represented as the number 5 in the new table.

Phone icon plus number: well turn off this option on the table and convert existing cells to Plain Text.

Flag: well turn off this option in the data table, and convert cells to plain text where we simply spell out the flag color (green, blue, etc.).

Multiple Choice: well turn off this option in the data table and render existing cells as Plain Text. We will also remove the ability to toggle to another choice. Existing Socrata forms powered by a multiple-choice drop-down will continue to work, but no new forms can be created with multiple-choice drop-downs.

You can read more about these deprecated column types .

So you want to publish your data online. You want it to look great, you want it to be useful and you want people to be able to search and filter it efficiently. Data types are critical to all of these functions, but how do you know when something is a valid type or not? How do you format your date column so it appears as a date? How do you get percentages to properly appear?

Read on, and let us know if you still have any questions!

Whichfile types are allowed on import?

Import a Data File:

.csv (csv output from an excel file or see rfc 4180 for standard format)

.xls (Excel 97 and later)

.xlsx (2007 and later)

.tsv

Note: .csv is the optimal file format for upload. If you experience an error while importing a .xls or .xlsx file, try saving as a .csv to eliminate Excel formatting that can sometimes cause errors.

Upload a Non-Data File:

The followin file types are allowed to be uploaded: accdb,. asc,. avi,. azw3,. bmp,. c,. css,. csv,. das,. dat,. db,. dbf,. dib,. doc,. docx,. dta,. dwg,. ecw,. eml,. fmw,. ged,. geojson,. gif,. gif,. gsheet,. ico,. ics,. indd,. jpeg,. jpg,. js,. json,. kml,. kmz,. log,. lyr,. md,. mdb,. mov,. mp3,. mp4,. mpg,. msg,. n3,. numbers,. ods,. odt,. one,. onepkg,. pages,. paint,. pb,. pdf,. png,. pps,. ppt,. pptx,. psd,. pxm,. rar,. rdf,. rtf,. sas,. sas7bcat,. sas7bdat,. sav,. sdf,. shp,. sql,. sig,. sketch,. snagproj,. tex,. tif,. tiff,. tmp,. tsv,. twbx,. txt,. wav,. wma,. wmf,. wmv,. wpd,. wps,. xlr,. xls,. xlsb,. xlsx,. xml,. xps,. zip (however we only provide previews for types listed in this article ). Due to Socrata security policies, any other file extension on this list will not be allowed to upload. It is possible to upload non-data files of at least 10 GB.

Import Geospatial Data:

.zip ( shapefiles ; the following files are required: .shp, .shx, .dbf, .prj)

.kml (We do not support multiple features in a single layer. Technical terminology: we do not support heterogeneous children inside a multi geometry tag)

.kmz (this a zipped .kml and follows the same import rules as above)

.json (GeoJSON)

Link to External Data:

URL

Connect with an ESRI Map Layer:

URL to the RESTful endpoint for an individual map layer on an ArcGIS Server version 10.0 or above. You can read more about these connections here.

In addition to ESRI Extension, Socrata has built out the Catalog Connector for ArcGIS Server. This new feature allows administrators to connect a public ArcGIS Server to your Socrata catalog and choose the layers to federate to your Socrata catalog as external datasets.

Our platform supports Web Mercator projections via ESRI Extensions and will not convert other projections types.

Supported Types on Import

We only try to detect specifically supported types on import. There are many other data types that we support but you'll have to either convert your columns after import or enter the data manually.

The following types are supported:

Plain Text

UTF-8 encoded text (generally alphanumeric text). Our system assumes that there is no text formatting.

Numbers

Numbers: We directly use Java's BigDecimal parsing. For details see the documentation. If you're working with values that contain leading zeros that aren't importing correctly, visit this support article.

For negative numbers, the format shouldnotcontain any commas.-10000will be read correctly but-10,000will not.

You can also upload data as a percent or as money. These values will be uploaded as number columns with formatting applied so that they display as percents or money. The display can be changed by modifying the column formatting.

Dates & Time / Date & Time (with timezone)

Dates are parsed by default in the American/Pacific (PST) timezone. You can explicitly specify a time zone by using the supported ISO 8601 subset. A 'Z' character is UTC, otherwise, the offset is[+-]HH:mm.

For inputs that don't specify a time of date, the resulting time is undefined. In other words, don't rely on it being anything consistent.

The accepted input formats are:

Supported ISO 8601 Subset

yyyy-MM-dd['T']HH:mm:ssZ(e.g. "1920-01-22T00:00:00Z", "1920-01-22T00:00:00-10:00", or "1920-01-22 00:00:00Z")

yyyy-MM-dd['T']HH:mm:ss(e.g. "1920-01-22T00:00:00" or "1920-01-22 00:00:00")

yyyy-MM-dd['T']HH:mm(e.g. "1920-01-22T00:00")

yyyy-MM-dd (e.g. "1920-01-22")

Supported non-ISO Dates

For dates other than the ISO subset we accept a date, optionally followed by a time, i.e.

(date)[ (time)]

Non-ISO dates are always parsed in the American date format locale (i.e. month, day, year). Months and days can be either single or double digit and may or may not be led with a '0'. Years can be either four digits (preferred) or two. If a year is two digits it will be assumed to be between 1951 and 2050: i.e.1/2/75would be January 2nd 1975, but1/2/49would be January 2nd 2049.

The accepted input formats are:

MMM d, yyyy(e.g. "Jan 4, 1982")

MMM d, yy(e.g. "Jan 4, 82")

MMMM d, yyyy(e.g. "January 4, 1982")

MMMM d, yy(e.g. "January 4, 82")

M-d-yyyy(e.g. "1-4-1982")

M/d/yyyy(e.g. "1/4/1982")

M.d.yyyy(e.g. "1.4.1982")

M-d-yy(e.g. "1-4-82")

M/d/yy(e.g. "1/4/82")

M.d.yy(e.g. "1.4.82")

Location and Point Columns

Datasets on Socrata can store location in two different data types Location and Point. In the future, all datasets will use the point format.

Both location and point columns can be built "composite" column that's created by appending multiple values together. We accept the following types of composite location data for geolocation:

Street Address (U.S. only)

City (U.S. only)

State (U.S. only, use the two-letter abbreviation)

Zip Code (both 5 and 9 digits are accepted; U.S. only)

Latitude and Longitude (All Locations)

In addition, you can upload location data to both location and point columns when the data is contained within an entire cell.

For location columns the format will need to be: (lat, long)

For point columns the format should be: POINT (long lat)

URLs

URL's support threedifferent input formats. Only three URL schemes are acceptable:FTP,HTTP, andHTTPS. We use a custom regular expression to validate URLs. It should accept just about anything that you throw at it, but there's always a chance that it's missed something. Please note that since underscores are invalid characters in DNS records, we do not accept underscores.Also, note that only one URL can be entered in a cell.

<a href=" http://www.socrata.com/" >Socrata</a>

http://www.socrata.com/

Socrata (http://www.socrata.com)

Checkboxes

Valid false values:

0

f

false

n

no

off

Valid true values:

1

t

true

y

yes

on

Whichdata types are available after import?

There are some file types that we do not handle during import, but you can add to your dataset after you have imported the file. It is also not possible to programmatically import these data types.

Photo (Image)

Accepted file formats: .jpg, .png, .gif

Document

The following file types are allowed to be uploaded: accdb,. asc,. avi,. azw3,. bmp,. c,. css,. csv,. das,. dat,. db,. dbf,. dib,. doc,. docx,. dta,. dwg,. ecw,. eml,. fmw,. ged,. geojson,. gif,. gif,. gsheet,. ico,. ics,. indd,. jpeg,. jpg,. js,. json,. kml,. kmz,. log,. lyr,. md,. mdb,. mov,. mp3,. mp4,. mpg,. msg,. n3,. numbers,. ods,. odt,. one,. onepkg,. pages,. paint,. pb,. pdf,. png,. pps,. ppt,. pptx,. psd,. pxm,. rar,. rdf,. rtf,. sas,. sas7bcat,. sas7bdat,. sav,. sdf,. shp,. sql,. sig,. sketch,. snagproj,. tex,. tif,. tiff,. tmp,. tsv,. twbx,. txt,. wav,. wma,. wmf,. wmv,. wpd,. wps,. xlr,. xls,. xlsb,. xlsx,. xml,. xps,. zip.

NOTE: FOR DATASETS THAT HAVE DEPRECATED FEATURES, THE FEATURES WILL STILL FUNCTION. HOWEVER, THESE FEATURES WILL NOT BE AVAILABLE FOR ANY NEW DATASETS.

Highlights

Socrata Gateway, a new framework for creating and automating datasets in Socrata, is now available to all customers. Please see support articles for getting started.

Users can now create a calendar visualization to display basic event information at a monthly view.***

The creation of reports through Report Builder has been disabled.Reports will remain accessible as read-only.

Minor Updates

Administration & User Management

Resolved an issue where users with proper permissions were not allowed to remove members from a team.

Resolved an issue where email notifications included an invalid link to update email preferences.

Resolved an issue in the user experience for adding users to teams. Users who are not allowed to be on teams no longer show in the autocomplete when searching for users on the team management page.

Catalog & Metadata

Added notifications to catalog when the number of assets returned exceeds the 10,000 result limit.

Resolved an issue with primer page layouts.

Resolved an issue where derived views were not being immediately granted public visibility after approval.

Data Ingress & Updates

Improved error messaging when date/time value is '0000-01-01 00:00:00.000', which is not an acceptable value.

Resolved an issue causing errors with plugins that included boolean user parameters.

Visualizations

Resolved an issue where icon configurations were being removed when editing a map.

Resolved an issue where users were unable to specify flyout unit labels on grouped charts.

Resolved an issue where the hierarchy dropdown on drilldown enabled charts was not auto-collapsing upon selection.

Coming Soon!

Expected release - 2 weeks

Administrators will have the ability to configure custom colors and palettes to be used by end-users on a domain-wide basis for creating visualizations.***

Expected release - 1 month

Users will soon have more advanced alert scheduling and threshold triggering options when creating an alert for a dataset.***

Updated Asset Inventory datasets will be added to all domains with schema changes adding Approvals status, visibility status, and row and column count for each dataset. Existing Asset Inventory datasets will continue to be updated until February 29th, 2020.

Legend: *** client-requested functionality

Curious about other areas of the platform? Visit our Performance or SCGC release notes for more! Or give us a shout at [email protected] with questions or ideas.

You must have Java 8 or higher installed in order to install an agent to connect directly to your data sources.

To provision and install a Socrata Gateway agent:

Selectthe gear icon, thenselect "Administration."

Select"Gateway."

Name the agent and download it.

Unzip the downloaded file to the location where you would like to keep the agent and open the new folder.

Open the macOS folder.

Double-click the setup file. If a warning comes up, select the option to open the file anyway. This will open your terminal.

Once the process in your terminal is complete, return to your Socrata site and click"Next."

Scroll down to the bottom of the Gateway modal and click "Refresh."

Refresh the page and see your new agent in the list of agents.

NOTE: For more information about Socrata Gateway agents, see Gateway Overview.

In order to run Gateway Agents, you'll need to make sure that you have installed the appropriate version the Java SE Development Kit (JDK) installed on your agent's system. Without the right version, JDK 11, you may run into the following error when installing the Gateway Agent on your Windows system.

Error Messages

[2019-11-06 14:47:44] [info] [ 8772] Commons Daemon procrun (1.1.0.0 64-bit) started[2019-11-06 14:47:44] [info] [ 8772] Service GatewayDemoDomainAgent name GatewayDemoDomainAgent[2019-11-06 14:47:44] [info] [ 8772] Service 'GatewayDemoDomainAgent' installed[2019-11-06 14:47:44] [info] [ 8772] Commons Daemon procrun finished[2019-11-06 14:47:45] [info] [ 1704] Commons Daemon procrun (1.1.0.0 64-bit) started[2019-11-06 14:47:45] [info] [ 1704] Running 'GatewayDemoDomainAgent' Service...[2019-11-06 14:47:45] [info] [ 9192] Starting service...[2019-11-06 14:47:45] [error] [ 9192] Failed creating Java [2019-11-06 14:47:45] [error] [ 9192] ServiceStart returned 1[2019-11-06 14:47:45] [info] [ 1704] Run service finished.[2019-11-06 14:47:45] [info] [ 1704] Commons Daemon procrun finished

Cause

ProcRun has failed to install. ProcRun is a Apache utility which allows us to run java applications as Windows Services. This is caused by either an incompatible version of java or an inability of procrun to verify which version of java is running on the machine.

Resolution

1. Download the OpenJDK 11 from https://adoptopenjdk.net/releases.html?variant=openjdk11&jvmVariant=hotspot (download the runtime version of java that machines your machine, most likely x64).

2. Extract the downloaded zip file into thewindowsdirectory of the agent.

3. Open the Agent EXE file (it will be named after the service that you attempted to create, in this case,it is called GatewayDemoDomainAgent).

4. Once it opens (it may ask for administrator privileges), go to the JAVA tab and select the ... icon to select a java virtual machine.

5. Navigate to the OpenJDK directory in your windows directory and select theserverfolder followed byjvm.dll - this will tell procrun which JVM version it should be using.

6. Select theAPPLYbutton.

7. Navigate back to theGeneraltab and select theStart button. The agent should now start running.

Socrata Gateway isan easy-to-use solution for database analysts and data publishers within the customer's organization to facilitate the ingress and scheduled updates of data from key on-prem and cloud-hosted source systems.

Terminology

Agent:An agent is a thin Java client that lives behind the users firewall that connects via WebSockets to Socrata backend systems. The client jar is available to download from the Dataset Management Page on Socrata and requires correct user authentication and correct permissions.

Plugin:A plugin is anextension to the agent that connects to specific data sources. For example, a Microsoft SQL Server plugin will contain the necessary drivers to connect to Microsoft SQL Server and can be configured to connect to a specific server and receive input from the agent owner on Socrata. In addition to reading local files and databases, plugins will be enabled to connect to API endpoints from various sources. Plugins are authored and released by Socrata and Tyler employees.

Hardware Specs

16 GB RAM

5 GB HD

The agent itself is ~15mb unzipped. Plugins vary between 10mb and 30mb. Data processing requirements will vary.

Supported OSs

Windows Server R2008

Windows 10

macOS Sierra +

Ubuntu 16, 18

Software

Java 8 + or

OpenJDK 11

Networking

The customer Socrata domain is the only required whitelisted URL.

Proxies

Proxies are supported natively. The agent will use the default system proxy that has been configured.

SSL

Special SSL certs must be added to the Java trust store. See: http://www.thesqlreport.com/?p=576

Internet Access

Outbound internet access via Port 443 required

Websocket and HTTPS communication, over SSL will be used for network communication

Security

All the credentials needed to access the on-premise source system, which resides in the configuration file of the agent, resides behind the firewall and does not get exposed to the Socrata platform.

The agent and its config file sit behind the user's firewall. The agent differs in that, it can talk to a users source systems.Only users with an account on Socrata with the correct access permissions can download the Gateway agent. Since the client needs to be configured to talk to the users source systems, users with special roles (eg: Data Administrators) will have access.

There is a change to code-serving, as the agent and plugins are dynamically loaded by the agent. Risks are mitigated by checking the Sha256 digest of the reportedly released version against the Sha of the version that was actually downloaded. If an attacker gained access to the S3 bucket where the agent or plugin .jar files reside and overwrote the actual version with theirs, the agent would reject the downloaded version. In order to release code, the attacker would need write access to the S3 bucket in addition to write access to the DSMAPI database, which are controlled by different keys and are on different networks. This is strictly more secure than the current release process of Datasync, which requires the user manually check the digest of the .jar file they download from the Socrata site, which people rarely, if ever, verify.

There is a change to the security attack surface, in that, if you are logged in and know the unique identifier of the agent, you could theoretically get it to a dataset that you have access to. This is mitigated by prohibiting agents to send data across domains, as well as making the agent identifier a UUID which is not possible to guess.

Architecture Diagram:

Permissions Requirements:

Windows: Administrator rights to set up a Windows Service

macOS: Administrator rights to configure service

Linux: Sudo access to configure service

Misc:

The installation file cannot be placed in a networked drive

Updates happen on an ad hoc basis in accordance with our bi-weekly release cycle.

This page describes the different types of events you may see in the activity log pertaining to setting up Gateway and Schedules.

Managing Agents and Plugins

Managing Schedules

Schedule Failures

Managing Agents and Plugins

Agent Provisioned

Agent Deleted

Plugin Configured

(Note the Item_Affected columns displays the name of the agent the plugin was configured under)

Plugin Deleted

(Note the Item_Affected columns displays the name of the agent the plugin was configured under)

Managing Schedules

Schedule successfully runs generates 4 records

Pausing and resuming schedule

Schedule Failures

Schedules and updates can fail, here are some common failures when using Gateway and Schedules.

Schedule failed due to source not existing

(e.g. changed the file name) generates sets of "Draft Created" and "Schedule Failed". The schedule will retry 35 times before pausing.

Note: This error will also appear on the dataset's Primer page.

Schedule update that succeeds but also includes records that don't import

Errors due to a bad data type (e.g. text in a date field) generate the following 4 records.

Schedule failure due to changing schema

e.g. deleted a column generates sets each of "Draft Created", "Draft Deleted" and "Data Update Import Failure"

URL schedule failed due to asset not existing

Occurs when URL import is loading on a URL that no longer exists, generates the following: "Draft Created" and "Schedule Failed". The schedulewill retry 35 times before pausing.

Schedule failed due to throttling

Schedules that run more than once a day and exceed the 2 million record limit in a 24 hr period, will retry 35 times before pausing.

Below is a list of known issues with Socrata Gateways specific to different plugins.

If you have questions, need help, or if you'd like to be notified when the issue is fully resolved, please reach out to us! Just email us at [email protected] and we'll get back to you soon!

Item Affected

Issue Description

Status

Excel Plugin

The Excel plugin cannot connect to files if the plugin is configured using a mapped drive location

Under Investigation

Excel Plugin

The Excel plugin times out when connecting to large (not yet quantified) files and does not provide an error message.

Under Investigation

Access Plugin

The Access plugin throws an error when trying to ingressing a blob type.

Under Investigation

MacOS Plugins

On restart, a plugin setup on MacOS doesn't automatically restart itself and connect to the relevant agent on the domain.

Fix in Progress

MySQL plugin

When setting up theMySQL plugin you specify the database. However, by default, the plugin will enable data publishers to see all tables and views based on the user'sMySQL permissions. When a data publisher attempts to select a table or view, not within the database configured by the plugin, they receive an error.

Fix in Progress

All Plugins

When adding, reconfiguring or deleting plugins from the command line, the plugin type and plugin name values are case sensitive. All plugin types are lower case.

Under Investigation

Let us know how we can help! Email us at [email protected]

Table of Contents

Overview

System Requirements

Set Up Instructions

STEP 1: Create an Agent

Starting the Agent

Managing the Agent

STEP 2: Add and Configure a Plugin

Option 1: Start from the Command Line

Option 2: Start from the Socrata site

Reconfigure a plugin

Removing a plugin

Overview

Establishing a Socrata Gateway connection is a two-step process:

Create an agent

Add a plugin (or plugins) to that agent

Agents are small programs that establish a secure connection to Socrata and act as a manager for a plugin or set of plugins.

After the agent is downloaded and set up, well help you connect to your individual data sources with plugins for specific data types.

You can add as many plugins as you like to a single agent, including multiple plugins of the same type.

System Requirements

The agent requires Java 8 or newer

Install Java from your system package manager

Debian/Ubuntu: sudo apt install openjdk-11-jre

Red Hat or CentOS: sudo yum install java-11-openjdk

Networking The host must be allowed to make connections to Socrata on port 443

SSLSpecial SSL and Self-Signed Certificates must be added to the Java trust store. For more information, please see: http://www.thesqlreport.com/?p=576

User Permissions Sudo (root) or Administrator access is required to configure the service

Set Up Instructions

STEP 1: Create an Agent

Starting the Agent

From your Socrata site click the Provision Agent button on either the:

Admin/Gateway page ({Your_Domain}/admin/gateway) OR

When you're choosing your data source while creating or editing a dataset

Follow the agent naming and download process.

Place the downloaded folder in a directory on the server or computer you are connecting to Socrata (if its not there already).Note or copy this folder path. Youll need it when you set up a plugin later.

Unzip the downloaded file.

Run `linux/install` in that directory. It will ask a few questions about where and how to run the service, and then set it up to be run automatically by `systemd`.

$ linux/install systemd service name: example User to run as: myuser Directory to run in: /home/myuser/example-agent Data directory: /home/myuser/.local/share/socrata-ingress-agent Here is the unit file we will generate: [Unit] Description=The Socrata Ingress Agent StartLimitIntervalSec=1 StartLimitBurst=5 [Service] User=myuser Group=myuser WorkingDirectory=/home/myuser/example-agent ExecStart=/usr/bin/java -jar "/home/myuser/example-agent/example.jar" --data-dir "/home/myuser/.local/share/socrata-ingress-agent" Restart=on-failure RestartSec=1 PrivateTmp=true PrivateDevices=true [Install] WantedBy=multi-user.target Is this correct? [Y/n] Y created /home/myuser/Downloads/gateway_example/example.service created /home/myuser/Downloads/gateway_example/example-control Setting up the service. The commands being run will be displayed. You may be asked for the root password, depending on your local sudo configuration. sudo mkdir -m0755 -p -- /home/myuser/example-agent sudo cp -- /home/myuser/Downloads/gateway_example/socrata-ingress-agent.jar /home/myuser/example-agent/example.jar sudo chmod 644 -- /home/myuser/example-agent/example.jar sudo mkdir -m0700 -p -- /home/myuser/.local/share/socrata-ingress-agent sudo chown -- myuser:myuser /home/myuser/.local/share/socrata-ingress-agent sudo cp -- /home/myuser/Downloads/gateway_example/example.service /etc/systemd/system/example.service sudo systemctl enable -- example.service Created symlink /etc/systemd/system/multi-user.target.wants/example.service /etc/systemd/system/example.service. sudo systemctl start -- example.service Setup complete. You may use the /home/myuser/Downloads/gateway_example/example-control script to control, configure, or uninstall the service.

On your Socrata site you can check the agent status in the following places:

In the Provision Agent set up stepper (On the Set up instructions page there is an Am I connected? widget. Clicking refresh will check your connection.

On the Admin/Gateway page ({Your_Domain}/admin/gateway), it will appear in the table

When you're choosing your data source while creating or editing a dataset it will appear in the list as Connected

Managing the Agent

After installing the agent, you can use the generated control scriptto manage it:

$ ./example-control Usage: ./example-control stop|start|status|logs|run|uninstall stop - stop the service Important: All plugins associated with this agent will also stop and any schedules associated with datasets created under this agent will stop/break start - start the service status - display information about the service logs - show the service's logs -f, --follow output appended data as more is generated run ARGUMENTS - run a command on the service jar Use "./example-control run --help" for more information uninstall - stop the service and delete all its data files $ ./example-control start $ ./example-control stop $ ./example-control status

STEP 2: Add and Configure a Plugin

TIPS:

PLUGIN_TYPE is a type from a list youll be presented upon set up.

PLUGIN_NAME is a name you select to identify this plugin on Socrata.

Configuration for the plugin type and plugin name are case sensitive. Please ensure youre using the case presented at set up.

You can add as many plugins as you like to a single agent, including multiple plugins of the same type.

Option 1: Start from the Command Line

Open the command prompt on your computer or server and navigate to the directory where the downloaded agent jar file resides.

To list available plugin types

$ ./example-control run --list-available-plugins

To install or reconfigure a plugin

$ ./example-control run --configure-plugin PLUGIN_TYPE:PLUGIN_NAME

When the plugin is run it will display a window on your server asking for plugin-specific configuration and credentials (if needed). This information will not live on Socrata.

Fill out the configuration information and click OK. Return to Socrata and your plugin will then be ready to use!

Option 2: Start from the Socrata site

Click "Add Plugin" either

On the Admin/Gateway page ({Your_Domain}/admin/gateway)

When you're choosing your data source while creating or editing a dataset

Choose your plugin.

Name your plugin.

Naming tip: Who is responsible + Data source description + Data source type

Example: Finance_Risk_CSV

Non alphanumeric characters will be converted to underscores

Open the command prompt on your computer or server and navigate to the directory where the downloaded agent jar file resides.

Copy and paste the plugin command into the command prompt.

Run the command by hitting enter or return. This will download the plugin from Socrata, verify it, and run it on your server. Tip: If you get an error, try changing directories.

When the plugin is run it will display a window on your server asking for plugin-specific configuration and credentials (if needed). This information will not live on Socrata.

Fill out the configuration information and click OK. Return to Socrata and your plugin will then be ready to use!

Reconfigure a plugin

On the computer or server your plugin has been configured, in the command prompt enter:

$ ./example-control run --configure-plugin PLUGIN_TYPE:PLUGIN_NAME

When the plugin is run it will display a window on your server asking for plugin-specific configuration and credentials (if needed). This information will not live on Socrata.

Fill out the configuration information and click OK. Return to Socrata and your reconfigured plugin will then be ready to use!

Removing a plugin

Important: All schedules associated with the dataset(s) created under this plugin will break when you remove the plugin.

On the computer or server your plugin has been configured, in the command prompt enter:

$ ./example-control run --delete-plugin PLUGIN_TYPE:PLUGIN_NAME

Table of Contents

Overview

System Requirements

Set Up Instructions

STEP 1: Create an Agent

Start the agent and run it as a service

Stopping the agent/service

Re-starting the agent

Delete or Uninstall the agent/service

Check the status of the agent on your computer or server

STEP 2: Add and Configure a Plugin

Option 1: Start from the Command Line

Option 2: Start from the Socrata site

Reconfigure a plugin

Removing a plugin

Overview

Establishing a Socrata Gateway connection is a two-step process:

Create an agent

Add a plugin (or plugins) to that agent

Agents are small programs that establish a secure connection to Socrata and act as a manager for a plugin or set of plugins.

After the agent is downloaded and set up, well help you connect to your individual data sources with plugins for specific data types.

You can add as many plugins as you like to a single agent, including multiple plugins of the same type.

You can find a 60-second video on setting up an agent in Windows here.

System Requirements

The agent requires Java 8 or newer Find it here: OpenJDK 11

Networking The host must be allowed to make connections to Socrata on port 443

SSL If you have special SSL and/or Self-Signed Certificates configured, they must be added to the Java trust store. For more information, please see: http://www.thesqlreport.com/?p=576

User Permissions Sudo (root) or Administrator access is required to configure the service

Set Up Instructions

STEP 1: Create an Agent

Start the agent and run it as a service

From your Socrata site click the Provision Agent button on either the:

Admin/Gateway page ({Your_Domain}/admin/gateway) OR

When you're choosing your data source while creating or editing a dataset

Follow the agent naming and download process.

Place the downloaded folder in a directory on the server or computer you are connecting to Socrata (if its not there already).Note or copy this folder path. Youll need it when you set up a plugin later.

Extract/Unzip the downloaded file.

Open the subfolder named Windows.

Run the agent as a service.

Right-click on the Install Windows batch file.

Choose Run as an Administrator.

This will launch the command prompt.

In the command prompt, name the Service (it will say Enter Service Name) and choose a name that makes sense to you.

Hit Enter.

Check the status of your agent to ensure the set up was successful. A successfully connected agent is needed before you can move on to step 2 of the Gateway set up adding and configuring a plugin.

On your Socrata site you can check the agent status in the following places:

In the Provision Agent set up stepper (On the Set up instructions page there is an Am I connected? widget. Clicking refresh will check your connection.

On the Admin/Gateway page ({Your_Domain}/admin/gateway), it will appear in the table

When you're choosing your data source while creating or editing a dataset it will appear in the list as Connected

Stopping the agent/service

Important: All plugins associated with this agent will also stop and any schedules associated with datasets created under this agent will stop.

From the start menu, search for and select "Services.

Scroll to your Windows Service (created in the Starting Your Agent section). The description will be 'The Socrata Ingress Agent.

Right-click on the service and select "Stop.

Re-starting the agent

Right-click the service and select "Start.

Delete or Uninstall the agent/service

Locally:

Open command prompt as an administrator.

Run: sc.exe delete {the name of your service}.

On Socrata: Note: This does not delete the service from your machine. It only deletes it from appearing on Socrata.

Navigate to {Your_Domain}/admin/gateway

Choose "Delete agent" from the "Actions" menu.

Check the status of the agent on your computer or server

Open the Windows Services Manager.

Search for Windows Service Manager OR

Run services.msc from the Run box

Find your agent. The description will be The Socrata Ingress Agent.

Check the Status column

STEP 2: Add and Configure a Plugin

TIPS:

PLUGIN_TYPE is a type from a list youll be presented upon set up.

PLUGIN_NAME is a name you select to identify this plugin on Socrata.

Configuration for the plugin type and plugin name are case sensitive. Please ensure youre using the case presented at set up.

You can add as many plugins as you like to a single agent, including multiple plugins of the same type.

Option 1: Start from the Command Line

Open the command prompt on your computer or server and navigate to the directory where the downloaded agent jar file resides.

To list available plugin types

java -jar socrata-ingress-agent.jar --list-available-plugins

To install or reconfigure a plugin

java -jar socrata-ingress-agent.jar --configure-plugin PLUGIN_TYPE:PLUGIN_NAME

Example: java -jar socrata-ingress-agent.jar --configure-plugin csv:Finance_Risk_CSV

When the plugin is run it will display a window on your server asking for plugin-specific configuration and credentials (if needed). This information will not live on Socrata.

Fill out the configuration information and click OK. Return to Socrata and your plugin will then be ready to use!

Option 2: Start from the Socrata site

Click "Add Plugin" either

On the Admin/Gateway page ({Your_Domain}/admin/gateway)

When you're choosing your data source while creating or editing a dataset

Choose your plugin.

Name your plugin.

Naming tip: Who is responsible + Data source description + Data source type

Example: Finance_Risk_CSV

Non alphanumeric characters will be converted to underscores

Open the command prompt on your computer or server and navigate to the directory where the downloaded agent jar file resides.

Copy and paste the plugin command into the command prompt.

Run the command by hitting enter or return. This will download the plugin from Socrata, verify it, and run it on your server. Tip: If you get an error, try changing directories.

When the plugin is run it will display a window on your server asking for plugin-specific configuration and credentials (if needed). This information will not live on Socrata.

Fill out the configuration information and click OK. Return to Socrata and your plugin will then be ready to use!

Reconfigure a plugin

Open the command prompt on your computer or server and navigate to the directory where the downloaded agent jar file resides.

In the command prompt enter:

java -jar socrata-ingress-agent.jar --configure-plugin PLUGIN_TYPE:PLUGIN_NAME

Example: java -jar socrata-ingress-agent.jar --configure-plugin csv:Finance_Risk_CSV

When the plugin is run it will display a window on your server asking for plugin-specific configuration and credentials (if needed). This information will not live on Socrata.

Fill out the configuration information and click OK. Return to Socrata and your reconfigured plugin will then be ready to use!

Removing a plugin

Important: All schedules associated with the dataset(s) created under this plugin will break when you remove the plugin.

Open the command prompt on your computer or server and navigate to the directory where the downloaded agent jar file resides.

In the command prompt enter:

java -jar socrata-ingress-agent.jar --delete-plugin PLUGIN_TYPE:PLUGIN_NAME Example: java -jar socrata-ingress-agent.jar --delete-plugin csv:Finance_Risk_CSV

Table of Contents

Overview

System Requirements

Set Up Instructions

STEP 1: Create an Agent

Start the agent and run it as a service

Stopping/deleting the agent

Re-starting the agent

Check the status of the agent

STEP 2: Add and Configure a Plugin

Option 1: Start from the download folder

Option 2: Start from the Socrata site

Reconfigure a plugin

Removing a plugin

Overview

Establishing a Socrata Gateway connection is a two-step process:

Create an agent

Add a plugin (or plugins) to that agent

Agents are small programs that establish a secure connection to Socrata and act as a manager for a plugin or set of plugins.

After the agent is downloaded and set up, well help you connect to your individual data sources with plugins for specific data types.

You can add as many plugins as you like to a single agent, including multiple plugins of the same type.

You can find a 60-second video on setting up an agent on a Mac here.

System Requirements

The agent requires Java 8 or newer Find it here: OpenJDK 11

Networking The host must be allowed to make connections to Socrata on port 443

SSLIf you have special SSL and/or Self-Signed Certificates configured, they must be added to the Java trust store. For more information, please see: http://www.thesqlreport.com/?p=576

User Permissions Sudo (root) or Administrator access is required to configure the service

Set Up Instructions

STEP 1: Create an Agent

Start the agent and run it as a service

From your Socrata site click the Provision Agent button on either the:

Admin/Gateway page ({Your_Domain}/admin/gateway) OR

When you're choosing your data source while creating or editing a dataset

Follow the agent naming and download process.

Place the downloaded folder in a directory on the server or computer you are connecting to Socrata (if its not there already).Note or copy this folder path. Youll need it when you set up a plugin later.

Unzip the downloaded file.

Run the agent as a service.

In the downloaded folder, open the macOS folder.

Double click on the setup file.

This will open the Terminal, set up the agent to run as a service, and complete the setup. Note: You can use this process to manually restart the agent.

Check the status of your agent to ensure the set up was successful. A successfully connected agent is needed before you can move on to step 2 of the Gateway set up adding and configuring a plugin.

On your Socrata site you can check the agent status in the following places:

In the Provision Agent set up stepper (On the Set up instructions page there is an Am I connected? widget. Clicking refresh will check your connection.

On the Admin/Gateway page ({Your_Domain}/admin/gateway), it will appear in the table

When you're choosing your data source while creating or editing a dataset it will appear in the list as Connected

Stopping/deleting the agent

Important: All plugins associated with this agent will also stop and any schedules associated with datasets created under this agent will stop.

Locally:

Navigate to the folder where the downloaded agent jar file resides.

In the macOS folder, double click the "stop" bash file.

On Socrata: Note: This does not delete the service from your machine. It only deletes it from appearing on Socrata.

Navigate to {Your_Domain}/admin/gateway

Choose "Delete agent" from the "Actions" menu.

Re-starting the agent

In the macOS folder, double click the setup bash file.

Check the status of the agent on your computer or server

Open the terminal on your computer or server.

In the terminal enter:

launchctl list | grep socrata

This command will list out all of the information on services running on your system. It will list the (PID) process id, status and service label for each service. The grep command will filter the list of services to display only those services that include socrata in the name.

A sample output will display:

1550 0 com.socrata.Gateway_SQL_Agent

If there are no results returned or no PID attached to the service then you may need to run the setup bash file again.

STEP 2: Add and Configure a Plugin

TIPS:

PLUGIN_TYPE is a type from a list youll be presented upon set up.

PLUGIN_NAME is a name you select to identify this plugin on Socrata.

Configuration for the plugin type and plugin name are case sensitive. Please ensure youre using the case presented at set up.

You can add as many plugins as you like to a single agent, including multiple plugins of the same type.

Option 1: Start from the download folder

Double click the "configure" bash script, follow the prompted instructions.

When the plugin is run it will display a window on your server asking for plugin-specific configuration and credentials (if needed). This information will not live on Socrata.

Fill out the configuration information and click OK. Return to Socrata and your plugin will then be ready to use!

Option 2: Start from the Socrata site

Click "Add Plugin" either

On the Admin/Gateway page ({Your_Domain}/admin/gateway)

When you're choosing your data source while creating or editing a dataset

Choose your plugin.

Name your plugin.

Naming tip: Who is responsible + Data source description + Data source type

Example: Finance_Risk_CSV

Non alphanumeric characters will be converted to underscores

Open the terminal on your computer or server and navigate to the directory where the downloaded agent jar file resides.

Copy and paste the plugin command into the command prompt/terminal.

Run the command by hitting enter or return. This will download the plugin from Socrata, verify it, and run it on your server. Tip: If you get an error, try changing directories.

When the plugin is run it will display a window on your server asking for plugin-specific configuration and credentials (if needed). This information will not live on Socrata.

Fill out the configuration information and click OK. Return to Socrata and your plugin will then be ready to use!

Reconfigure a plugin

From the download folder double click the "configure" bash script.

It will ask you to enter the plugin. Choose the TYPE of plugin you want to reconfigure (Note: Its case sensitive). Press enter [ENTER].

It will ask you to name your plugin. Type in the name of the existing plugin you want to reconfigure (Note: Its case sensitive). Press enter [ENTER].

When the plugin is run it will display a window on your server asking for plugin-specific configuration and credentials (if needed). This information will not live on Socrata.

Fill out the configuration information and click OK. Return to Socrata and your reconfigured plugin will then be ready to use!

Removing a plugin

Important: All schedules associated with the dataset(s) created under this plugin will break when you remove the plugin.

Open the terminal on your computer or server and navigate to the directory where the downloaded agent jar file resides.

In the terminal enter:

java -jar socrata-ingress-agent.jar --delete-plugin PLUGIN_TYPE:PLUGIN_NAME Example: java -jar socrata-ingress-agent.jar --delete-plugin csv:Finance_Risk_CSV

You must have Java 8 or higher installed in order to install an agent to connect directly to your data sources.

To provision and install a Socrata Gateway agent:

Selectthe gear icon, thenselect "Administration."

Select"Gateway."

Click "Provision Agent."

Name the agent and download it.

Extract the downloaded file to the location where you would like to keep the agent and open the new folder.

Open the windows folder.

Right-click the install file, and select "Run as administrator."If a warning comes up, select the option to run the file anyway. This will open your command line.

Type the name for your agent in the command line and hit"enter" on your keyboard.

Once the process in your terminal is complete, return to your Socrata site and click"Next."

Scroll down to the bottom of the Gateway modal and click "Refresh."

Refresh the page and see your new agent in the list of agents.

NOTE: For more information about Socrata Gateway agents, see Gateway Overview. If you run into any Java errors doing this, please see Gateway Java Errors.

You must have Java 8 or higher installed in order to install an agent to connect directly to your data sources.

To configure a CSV plugin for an existing Socrata Gateway agent:

Selectthe gear icon, thenselect"Administration."

Select"Gateway."

Click"Add Plugin."

Namethe plugin andclick"Next."

Copythe script.

Openyour command prompt. You may do this by hitting the Windows key on your keyboard and typing "cmd".

Change directoriesto the directory where your agent is, thenpastethe script and hit enter. If you need help navigating your command prompt, view this YouTube video.

When Java opens,navigateto the folder with CSVs you would like to put on Socrata.

Click"Open", then"OK."

Returnto your Socrata site andclick"Done."

Refreshthe page and see your new plugin listed under the agent.

NOTE: For more information about Socrata Gateway agents, see Gateway Overview.

Calendar Visualizations are now available on all sites across the platform. Users can create a calendar visualization to display basic event information at a monthly view.

Requirements

A dataset or filtered view of a dataset with a date/time data type column. (Note: date/time values which are stored as text will not work for configuring this type of visualization.)

How to Create a Calendar

1. Open a new visualization on your dataset (step-by-step instructions for how to do so are here ).

2. When the visualization editor opens, the Calendar option will be on the end of the list of visualization types at the top:

Perspectives Stories

3. In the left-hand panel, navigate toData > Data Selection.

Start Date: Begin by selecting the date/time type column in your dataset that is appropriate for the Start Date value. This is a required field.

End Date: This field is not required, but can be configured if you have events in your dataset spanning more than one calendar date.

Event Title: Select a text type column which contains the descriptive title data for your events.

4. In the left-hand panel, navigate toPresentation. Use theGeneral settings here toconfigure your calendar's title, enter a brief description, and select basic options like whether to include link to the source dataset; then, use theColor settings to configure the background and text colors of the events on your calendar.

5. In the left-hand panel, navigate to Legends and Flyouts > Flyout Details. This is where you can customize the level of detail shown on the flyouts which appear when users hover their cursor over a given calendar event. Note that the default flyout will just show the Event Title, Start Date, and End Date (if configured).

6. Click Save Draft to save your changes and preview the result.

NOTE: There is a Reset button at the bottom left of the configuration panel of the visualization editor. This will completely erase all of your configuration settings if at any point you would like to start over. It is not an Undo button, so use with caution!

Export Options

When exporting calendar data, users have the option to download the entire data file of a dataset, or its filtered contents (if filters are applied on the calendar).

Calendar data can be exported in the following file formats:

CSV, CSV for Excel, CSV for Excel (Europe), RDF, RSS, TSV for Excel, and XML.

Embeds

Like the other visualization types on our platform, Calendar visualizations can beinserted into content blocks within, or embedded on other webpages. Click the share icon on any Calendar visualization to open the Share and Embed modal, where you can copy iFrame as well as JS Embed code, and view previews of your embed.

You must have Java 8 or higher installed in order to install an agent to connect directly to your data sources.

To configure a CSV plugin for an existing Socrata Gateway agent:

Selectthe gear icon, thenselect"Administration."

Select"Gateway."

Click"Add Plugin."

Namethe plugin andclick"Next."

Copythe script.

Openyour terminal. You may do this by hitting thecommand+space barkeys on your keyboard and typing "terminal".

Change directoriesto the directory where your agent is, thenpastethe script andhitenter. If you need help navigating your command prompt, view this YouTube video.

When Java opens,navigateto the folder with CSVs you would like to put on Socrata.

Click"Choose", then"OK."

Returnto your Socrata site andclick"Done."

Refreshthe page and see your new plugin listed under the agent.

NOTE: For more information about Socrata Gateway agents, see Gateway Overview.

Once you have imported a dataset, with a point (location) column, you can use the visualization tool to create point, heat, and region maps.

Accessing the Visualization Tool

From the landing page of your dataset, simply click on the Visualize button and select Create Visualization. Or, from within the dataset, click the green Visualize tab and click on Launch New Visualization. This will open a new page where you can begin building your visualization!

To get started, you'll want to click on the + Add Visualization button in the upper right corner of the page.

Creating Your Map

Once you do this, you'll open up the visualization editor. To create a Map, click on the globe icon to start the map creation process!

Once you have selected the globe icon, some details will immediately generate. To the left of the page, there are two menu buttons that can be used to customize your map visualization: Map Layers and Map Settings.

The Map Layer section will be where you configure, shape, and stylize the data that is used to generate the map. You can also view and select other layers to add to your map, you can find more information on creating multilayer maps here.

The Map setting section will be where you configure the behavior of the map across layers. Here you can adjust things such as the title, description, base map, and search options.

Map Layers

We will first explore all the options available to customize your point map in the Map Layers section.

Data Selection

The main component of your map can be found under Geo Column. When selecting the globe icon, this field will pre-populate with the location column. If your dataset has more than one location column, you can select an alternative from the list of columns below the dimension field. By default, the map will start as a point map, but it can be changed to either a heat or region map in the Point Aggregation section.

Resize Point By Value

This section allows you to choose a number column whose values you can use to resize the points on the map.

Style by Value

This section will allow you to modify the point colors based on the values of any column in your dataset. You can also set an icon to be used for any value and color code that icon.

Point Aggregation

In this section you have three options:

None - Keeps your map as a standard point map

Heat Map - Creates a heat map

Region Map - Creates a region map utilizing custom boundaries already created on your domain. Please note that region maps will not function correctly on datasets that have multiple location columns.

Select Custom Boundary - A dropdown populated with custom boundaries set up in Spatial Lens.

Measure - Define how your region map is measured, select either a count of rows in the dataset or measure any numerical column using Sum, Average, Median, Min, or Max.

Color

Region Maps

In region maps, you can select the color palette as well as the number of data classes the region map should be broken into.

You also have the option to select one of two Classification Methods.

Jenks - Existing default selection which you can read about here.

Linear - Simple linear regression breaking data classes into even distribution buckets based on the data in the dataset. The lower bound is inclusive whereas the upper bound is exclusive, so 504.61004.2 means "504.6up to but not including1004.2." However, for the last bucket (i.e., the highest values), the lower and upper bounds are inclusive, so in the example below, 2003.42503 means "2003.4up to and including2503."

Standard Point Maps

Choose a color or color palette (when coloring by value) for the points on your map. You can also choose the opacity level of the points themselves.

You can also add an icon to your map point by using the Icon tab and performing a keyword search to find the best icon from the available icon library.

Simplification Level

Choose between More Detail, Normal, and Less Detail. The level of detail can affect the performance of the map, with more complex data you may want to choose less detail in order to increase performance.

Point Size

Choose the size of your point or the size range (when scaling points by value). When choosing the size range you set both the minimum and maximum size of a point to be used in the map. You can then select a number in the dropdown for "Number of Data Classes". Each data class will represent a single point size in which a subset of your data will fall into. If you select five data classes, the map will have five possible point sizes and your data will be placed into a corresponding bucket based on its value.

Flyout Unit Label

Edd the singular and plural version of the unit label used when hovering over a point.

Flyout Details

When clicking on a specific point, these are the details that will be displayed. You can configure a column to be used as the Flyout Title as well as add additional columns to your flyout details.

Legend

A checkbox to show/hide the map legend.

Map Settings

In this next section, we can set a number of different map setting options.

General

Set the title and description of the map, this will show up in the map area but will not change the name the map is saved as. You can also toggle the "View Source Data" link on and off, this link will direct users back to the dataset used to create the map.

Basemap

In the Type drop-down, choose from a list of nineprepopulatedbase maps, the map will refresh with your selection.

Map Controls

In this section, you can choose wheater you want to show three different controls: the search bar, the location button, and the navigation button. The search bar allows you to search for a specific location, the location button will zoom to the point where you currently view the map, and the navigation buttons will allow users to zoom in and out.

Search Boundary

You can select a custom search boundary on the map by selecting shift + click to draw a box on the map that sets custom boundaries. These boundaries will bound your search results on the Map.

Clusters

Point maps are a useful and intuitive way to show where incidents are occurring in a given region. They do, however, have their limitations - specifically, when looking at dense data, looking at points when zoomed out, and when multiple points occur in the same location.

In these cases, Socrata uses two different point aggregation features to make point data more useful and intuitive - stacking and clustering. Socratas geospatial visualization tools allow users to configure these capabilities to ensure that the final visualization is useful and meaningful.

What are Clusters?

Points grouped together based on geographic proximity, represented by a circle with a number inside. Clusters show a high-level view of your point data when zoomed out. Clicking on a cluster zooms the user into a closer view of the points contained in that cluster.

What are Stacks?

When points are very close together or located at the exact same coordinates, we render them as stacks (small circle with a colored outline) rather than have them all overlap. Clicking on a stack spiderfies out the points so that you can easily interact with each one.

Cluster Settings

Stop Clustering at Zoom Level

At this zoom level, show individual points instead of clusters. 1=zoomed out; 22=zoomed in. Note: If your map contains a lot of data, we may render clusters at lower zoom levels than you set (for performance reasons).

Cluster Radius

Adjust the size of the area by which points are grouped together into clusters. A smaller radius will produce more clusters, and a larger radius will produce fewer.

Cluster Size

Make clusters look bigger or smaller. Note: If you have set a Resize by Value column, that data will dictate cluster size rather than this control.

Stack Radius

Adjust the size of the area by which points are grouped together into stacks. A smaller radius will render more individual points.

One of our core focuses is on fully equipping our customers to manage their solutions and grow their data accessibility programs. Your Support Program helps ensure continuous support as your data program evolves.

What is Coaching?

Technical Coaching is designed to help increase the value of your existing Socrata services, included as part of your Silver, Gold, or Platinum Support Program.Your Socrata Coach is your technical partner for a scheduled session. We'll start by reviewing and understanding your current and future focus areas. Then, you and your Coach will partner on a particular set of questions, issues, or technical roadblocks you might be encountering this could range from creating effective visualizations, setting up metadata, to configuring a data automation workflow.

How Does a Coaching Session Work?

Our Coaching Sessions are usually self-contained one-hour engagements, conducted remotely, with up to three attendees. Our goal is to help develop and advance technical understanding as your open data program grows. To accomplish this, we'll partner with you to dive into your questions, a goal you have, or roadblock you're facing to help you strategize the most suitable option(s) based on your unique situation. Your Coach may demonstrate applicabletechniques, tools, or procedures with to help you achieve success.

Coaching Topic Examples

Data publishing

Data automation

Metadata setup and configuration

Roles and Permissions

Leveraging Site Analytics to inform focus areas

Open Finance dashboard configuration

Header and Footer Management

Administering Stories

Configuring a performance goal

Public Finance configuration

Setting up your Esri Connector

Federating content

Visualization building

Scheduling a Session

Customers with a Support Package of Silver, Gold or Platinum have access to a Socrata Consulting Coaches.To schedule a session either reach out to your Socrata Customer Success Manager (CSM) or send an email to [email protected] are typically scheduled one week in advance.

To help maximize your time, please include the following information:

The topic of the engagement

Specific goals or desired session outcomes

The Socrata site(s) to be used during the session along witha link to any specific datasets, if applicable

Relevant data files (csv)

Your preferred days or times

Session Cancellation Policy

We work hard to ensure that our team is well-prepared and focused on each engagement. If you need to reschedule your Coaching Session, please provide 48hrs notice by emailing [email protected]. If wedon'treceive adequate notice, we may deduct from your Coaching Hours half of the scheduled time.

More About Coaching

Learn more about our Support Programs and Policies at the following link: Support Policy.

When importing data, Socrata will automatically detect if your file can be parsed and uploaded in a tabular format. In the case that the file is non-parsable, it will be uploaded as a "File and Documents" asset that can only be previewed and downloaded by a user.

If you think your data should be parsed as tabular data and it is not, it is because your file does not have an extension listed in the supported 'parseable file formats' on the upload page. Currently, the supported parseable file formats are: .csv, .tsv, .xls, .xlsx, .zip (shapefile), .json (GeoJSON), .geojson, .kml, .kmz.

Socrata Data Specialist training is accelerated training tailored to your data. Learn how to work with data on Socrata from cleaning and preparation all the way to visualizing, shaping, and joining, alongside a group of your coworkers. Complete a capstone project and receive your certificate.

If you would like to learn more about becoming a certified Socrata Data Specialist, contact [email protected].

Certified Socrata Data Specialists

Shanna Brown, MPA

Research Analyst (DSRIP)

Austin Public Health

Alice Yuan, MSN/MPH, RN

Disease Surveillance Nurse

Austin Public Health

Stefan Wray, MPAff, MA

APH Open Data Program Lead, Research Analyst Senior

Austin Public Health IT

Flor Hernandez Ayala

Epidemiologist

Austin Public Health

Isaias Garcia

Vaccine Specialist

Austin Public Health

Maren Lujan

CHA/CHIP Planner

Austin Public Health

Warren Hendricks

Data Manager

Fulton County Department for HIV Elimination

When creating a visualization in the visualization canvas, you have the option to add filters via the quick filter bar.

Filtering a visualization will change the data that is in both the chart or map as well as the data table underneath.

Dependent Filters

When adding multiple column filters to a dataset, column filters are dependent on the results from the other filtered columns.For example, I add a filter for Neighborhood and include only Queen Anne. Then I add a second filter for Police Precinct. The second filter will only show precincts that operate in Queen Anne.

This is also useful when hiding a filter from end users; they will no longer be able to select additional values for data that does not exist in the dataset.

Add a filter

To add a filter to your visualization, select the Add Filter button located in the Filters panel on the right-hand side of the page. This will open up a panel where you can select any column in your dataset and use it as a filter.

create a filtered view

You can add as many filters to your visualization as you would like and configure each one in a way that works best.

Filter Options

There are a number of different options for each filter. These options can be accessed by selecting the vertical ellipsis to the right of the column.

Multiple Values vs. Single Value

This setting allows you to specify the number of values that a column can be filtered on. Setting this value to Multiple will allow users to choose a number of different text values, or a range of numerical values.

Setting it to Single value will limit you to a single text or numerical value.

Hidden vs. Interactive

Viewers of your visualization cannot add or remove columns to filter on. In addition, you can choose whether the filter that you have added is usable by the viewer.

When Interactiveviewerscan apply different values within the filters you have set up. You could also not apply any values to the filter at all, simply adding it for users to filter as they wish. Choosing Hidden will completely remove the filter from the published view.

Different types of filters

The operations you can perform on a filter vary based on the datatype of the column selected.

Text

Text columns values can be pulled in using an autocomplete function. Simply start typing your value and results from the dataset will appear in the filter window.

Selecting the dropdown next to the column name will bring up the list of functions available on text columns:is, is not, starts with,contains,does not contain.

Number

Numerical data can be filtered with a number of different operators. As seen below the full list of available operations are: exclude missing values, is equal to, is not equal to, is less than, is at most, is greater than, is at least, is between, is between and includes.

Location

You can also filter on a location column to show only points that fall within a certain radius. You can search by address to enter an address that would be found on the map, this bar will autocomplete to a list of available addresses.

From there You can use either the slider bar or the text box to choose the radius in miles to filter your results.

Region (Spatial Lens) Filtering

If you have a Spatial Lens configured on your site, you can use those boundaries to filter data in visualizations and maps built from datasets with a georeference column (i.e., location/point data). When creating a visualization based on an eligible dataset, you will see a "Region" option in the filter pane, which lists the Spatial Lens boundaries available for filtering. When you select a boundary, the data displayed on your visualization will be filtered down based on whether the location data falls within the selected boundary.

Dates

Dates can be filtered two different ways, by selecting a distinct range or choosing a relative date period.

Range

This option will allow you to select a distinct inclusive set of dates to filter your dataset. For example, this would be useful if you wanted to look at a single year.

Relative Date

Choosing a relative option will allow you to show a set number of completed periods based on the options you select. You can choose between days, months quarters, or years as well as some preselected options.

We have recently added the ability to also filter on quarters - which are defined by the calendar year and cannot be customized (i.e. Q1 = January-March, Q2 = Aril-June, etc).

The list of preselected options includes Today, Yesterday, This Week, This Month, This Quarter and This Year.

When using relative date filters, only completed time periods would be included. For example, let's say today is April 15, 2019.

Choosing the Last 5 days will show all data with a date/time between April 10-14th.

Choosing the Last 5 months will show all data from November 2018 - March 2019.

Choosing the Last 5 years will show all data from 2014-2018.

Choosing the Last 5 Quarters will show all Quarters from 2018 as well as Q1, 2019. So in total January 2018 - March 2019.

If you want to just show the last year April 15, 2018 - April 14, 2019, the option to choose would be theLast 365 days.

FAQ:

Q: Can I add multiple filters on the same column?

A: No you can only create filters once on a column. This most commonly comes up with numerical data. For instance, if you want to visualize two different ranges (i.e. 1-10 and 90-100). The best way to approach this is to of your dataset and base your visualization on that view.

Within Open Performance,users can specify the y-axis minimum and maximum values on the prevailing measure chart, helpingpublishers meetspecific goal visualization needs.

Lets walk through an example ...

Ensuring that you have first created a goal.The last step inthe goal chart set-upwizard is to preview the calculation and resulting line graph. This is wherewe'll begin.

To modify the y-axis, click the new button labeled Override Chart Scaling.

MAX and MIN fields appear. Simply deletethe existing numeric values and typeyour preferred minimum and maximum chart values. The chart will automatically resize itself according to your specifications. In our example, weve changed the default y-axis minimum of 6 to 0.

Click Finish in the lower right corner of the chart, then Saveat the top of the screen.

Your goal chart should now reflect the y-axis min and max values you just entered.

Minor Updates

Administration & User Management

Resolved an issue where users could not transfer ownership of community-created assets.

Resolved an issue where not all search results for users were returned when transferring ownership.

Resolved an issue where extra users were appearing in the autocompleted list when adding collaborators to assets.

Resolved an issue where the privacy policy acceptance text was misaligned on sites that offer multiple languages.

Catalog & Metadata

Resolved an issue where tags from federated assets did not appear in the tags facet of the catalog on the target domain.

Data Ingress & Updates

Modified Schedule Updates button on Primer page to make it more visible.

Resolved an issue where the restore function in the activity bar of a dataset draft displayed confusing images.

Resolved an issue causing errors when location fields in forms were populated.

Resolved an issue in which the "in between" filters on date type columns were not returning results.

Resolved an issue in which the "does not contain" filters were not returning results.

Visualizations

Resolved an issue where Y-axis labels were rendering partially obscured on charts in Firefox and Edge web browsers.

Coming Soon!

Expected release - 2 weeks

Socrata Gateway, a new framework for creating and automating datasets in Socrata by connecting directly to on-premise and cloud based systems, will be available for all customers.

Users will have the ability to create a calendar visualization to display basic event information at a monthly view.

Administrators will have the ability to configure custom colors and palettes to be used by end-users on a domain-wide basis for creating visualizations.

The creation of reports through Report Builder will be disabled by November 18, 2019. Reports will remain accessible but read-only.

Expected release - 1 month

Users will soon have more advanced alert scheduling and threshold triggering options when creating an alert for a dataset.

Legend: *** client-requested functionality

Curious about other areas of the platform? Visit our Performance or SCGC release notes for more! Or give us a shout at [email protected] with questions or ideas.

The SODA API Endpoint exists for all datasets, published versions and working copies. You can point to this reference using an application to consume from and/or publish to the dataset.

For all information on the different versions of API Endpoints offered, follow this link.

There are threeways you can access the API Endpoint:

1. Through the dataset URL:You can get the API endpoint by simply taking the dataset's UID and replacing it inthis string:

https://domain/resource/UID.extension*where theextension is the data format you's like to pull the data as.For a full list of extension formats please go here.

2.Through theAPIbutton on the dataset's Primer page.

this link

3. Through the Exportpanel when viewing the data: Click on View Data from the datasets primer page> Click on Export thenSODA APIto get to the API Endpoint

Note: Many datasets have two different endpoints, a SODA 2.0 and a 2.1 endpoint. Generally, the SODA 2.0 endpoint uses the UID you see when viewing the dataset online. This is the endpoint that should be used when publishing updates to the dataset.

The API URL in the export section of the dataset will typically use the SODA 2.1 endpoint. This is best used for consuming the data as it supports additional data types and functions.

Again, you may read more about the endpoints versions by followingon the Socrata Developer site.

What is data automation?

Data automation is the process of updating data on your open data portal programmatically, rather than manually. Automating the process of data uploading is important for the long-term sustainability of your open data program. Any data that is updated manually risks being delayed because it is one more task an individual has to do as part of the rest of their workload.

There are three common elements to data automation: Extract, Transform, and Load, or ETL.

Extract: the process of extracting your data from one or many sources systems

Transform: the process of transforming your data into the necessary structure, such as a flat file format like a CSV. This could also include things like changing all state abbreviations to the full state name.

Load: the process of loading the data into the final system, in this case the open data portal.

Each one of these processes is critical to fully automating your data uploads, and doing it successfully. Below is a diagram of a standard ETL process:

Pentaho ETL Toolkit for Socrata publishers

What is your data automation strategy?

Before diving into the technical nitty-gritty of data automation, it is important to determine a general data automation strategy for your organization. Having a strategy beforehand will help you engage the right people at the right time within your organization.

Who owns data automation in your organization?

Depending on your team model, different groups will own different parts of the ETL process:

Centralized: Central IT organization owns the full ETL process and all data automation.

Hybrid: This model may vary, but often the individual agencies/departments will own the extract and transform processes, and the central IT organization will own the loading process.

Decentralized: The individual agencies/departments will each own their own ETL process.

What tools are available for data automation?

Data automation tools vary in which process they are best suited, the technical knowledge needed to use them, and the cost. Below is a table outlining the most common tools used in automation:

E, T, or L?

Technical Knowledge

Cost

DataSync

L

Medium

Free

FME

E, T, and L

Medium

Yes

Pentaho Kettle

E and T

High

Free

Soda API

L

High

Free - but will require developer time for custom scripts

Beyond these tools, you can also employ a developer to create custom scripts for any of the E, T, or L processes. While this method provides more flexibility, it is more time and resource intensive.

How does automation fit into your publishing workflow?

After you have defined where the ETL stages will live within your organization, the next step is determining where your automation steps will fit into your publishing workflow. As you create your publishing flowchart, make sure to clarify the steps for Extract, Transform, and Load, as well as the individuals who will own those steps. Two of the key individuals who are likely to be involved with automation are the Data Custodian and the Data Steward.

What data should be automated?

As much data as possible! The more that you adopt an automate by default approach to uploading data, the less resources you will need long term for maintaining high data quality. Here are some tips for finding candidate datasets for automatic uploads:

Is the dataset updated quarterly or more frequently?

Are there transformations or any form of manipulation that needs to be done to the dataset prior to uploading?

Is the dataset large (greater than 250MB)?

Can you only get the changed rows for each subsequent update (rather than the full file)?

Is it possible to get data from the source system, rather than from an individual?

Datasets that prompt a yes to any of the questions above are great candidates for automating updates, because automation can remove the risk of a lack of time and resources later on.

Quick-start steps for automation

Once you understand the landscape of data automation within your organization, you can start putting your automation strategy to use. Here are some steps to get started:

Identify data: Select 1-2 high-value datasets where getting access to the source systems will be easy. (i.e. start with the low-hanging fruit)

Refer to your source system inventory to determine which source systems you already have access to.

Determine access: Determine how either the central IT organization or the department/agency will obtain the data. Will it be through a SQL query, download of a CSV, etc?

The Data Custodian will need to be involved in this step, as they are the best resource for accessing a datasets source system.

Define transformations: Outline any transformations required for that dataset. This might be as simple as changing complex acronyms to full-text names, or as complicated as transforming a relational database into a flat-CSV file.

Work with both the Data Steward and the Data Custodian to understand which fields need to be pulled and how they should be formatted for publication.

Develop and test ETL process: Based on the requirements defined in steps 2 and 3, select an ETL publishing tool, and publish the dataset to the Open Data Portal. Confirm that the dataset has been successfully loaded or updated through your process without errors.

Schedule: Schedule your dataset for timely updates.

Refer to metadata fields about data collection, refresh frequency, and update frequency you collected as part of your data inventory or dataset submission packet.

Technical resources for getting started

As you define your automation strategy and start working on automating those first few datasets, youll inevitably start getting into some of the technical details. Take a look at these guides to learn more about the different automation tools, and then speak with your Project Manager or Customer Success Manager to talk through next steps for getting your process implemented.

Getting Started with DataSync

Using FME to publish data to Socrata

Socrata datasets, including private datasets, can be accessed through a unique OData endpoint, allowing users to seamlessly connect to their data through a number of different tools.

As of 11/06/2017, Socrata's OData endpoints will also support OData v4, following the URL format of https://$domain/api/odata/v4/$dataset_identifier. If you are using the previous version (v2) of OData, endpoints follow this URL structure: https://$domain/OData.svc/$dataset_identifier, we recommend updating to the latest version.

Finding the OData Endpoint

You can easily locate the OData endpoint for a dataset on both the dataset's Primer page and through the data table.

To access the endpoint through Socrata Primer, select the "..." button on the top right of the screen. This will open up a drop-downwhere you can select "Access Data via OData". This will bring up a pop-up screen which will contain the OData Endpoint for the dataset. You can click on the OData box to choose either the v2 or the v4 endpoint.

http://www.odata.org/documentation/

This endpoint can also be found in the table view. When viewing the dataset select the Export pane and choose the OData section and copy either of the two listed OData endpoints.

Analyzing OData through Microsoft Excel

OData connections are often analyzed through Microsoft Excel. Supported versions of Excel and Power BI for use with Socrata OData are:

Power Query Version 2.40.4554.161 or newer - Excel 2010 / Excel 2013

Get & Transform Tool - Excel 2016 version 16.0.6965.2117 or newer

Power BI Desktop - Version 2.42.4611.901 or newer

For more information on using Excel to connect to your OData endpoint, please see the following article.

Authenticating in OData

In some cases, it is required to authenticate with your Socrata username and password to access an Odata feed. This can be done using thebasic authentication optionin most software that offers Odata connections.

The main cases for requiring authentication is:

When the dataset or view you are connecting to is Private.

When the dataset or view has been rejected in approvals from view in the catalog. Even though these assets are not private, since they are hidden from the catalog, connecting to these assets will require authentication. (Hint: You can tell if an asset is not approved when it can be viewed on your browser without logging in but the asset is not available in the Public Catalog).

Querying using OData

You can also directly query Socrata data using OData as a REST API. Socrata datasets follow the OData URI Conventions documented in https://www.odata.org/documentation/odata-version-2-0/uri-conventions/. For more information on OData, see

Note: Not all querying functionality is supported in V4 endpoints, please use V2 endpoints for data filtering.

Here are some examples of how to apply these conventions to a Socrata catalog and dataset:

List all the datasets available for a domain.

https://sandbox.demo.socrata.com/OData.svc/

Retrieve all the items in a particular dataset.

https://sandbox.demo.socrata.com/OData.svc/nimj-3ivp

Retrieve one item, by identifier

https://sandbox.demo.socrata.com/OData.svc/nimj-3ivp(16)

Retrieving a particular field within the item

https://sandbox.demo.socrata.com/OData.svc/nimj-3ivp(16)/depth/$value

The OrderBy Query option

https://sandbox.demo.socrata.com/OData.svc/nimj-3ivp?$orderby=magnitude

The Top Query option

https://sandbox.demo.socrata.com/OData.svc/nimj-3ivp?$top=2

The Skip Query Option

https://sandbox.demo.socrata.com/OData.svc/nimj-3ivp?$skip=1005

The Filter Query Option

https://sandbox.demo.socrata.com/OData.svc/nimj-3ivp?$filter=magnitude%20gt%205

The Select Query Option

https://sandbox.demo.socrata.com/OData.svc/nimj-3ivp?$select=magnitude,depth

The InlineCount Query Option

https://sandbox.demo.socrata.com/OData.svc/nimj-3ivp?$filter=magnitude%20gt%205&$inlinecount=allpages

Socrata Datatypes exposed as OData

This table lists the OData types used to represent Socrata data.

Socrata Datatype

ODataDatatype

Checkbox

Edm.Boolean

Date & Time

Edm.DateTimeOffset

Date & Time (with timezone)

Edm.DateTimeOffset

Document

Complext type with Edm.String listing the URL to the document and Edm.String listing the name of the document

Email

Edm.String

Location

Complex type with Edm.Decimal for latitude/longitude and Edm.String for Address, City, State and Zip

Money

Edm.Decimal

Multiple Choice

Edm.String

Number

Edm.Decimal

Percent

Edm.Decimal

Phone

Complext type with Edm.String listing the phone number and Edm.String listing the phone type

Photo

Edm.String listing the URL to the photo

Text (Formatted)

Edm.String with HTML tags

Text (Plain)

Edm.String

Website URL

Complex type with Edm.String listing the URL and Edm.String listing the description

Four data types are not exposed via OData. These are: Star, Flag, Nested Table and Dataset Link.

Interested in learning about our new features? Subscribe to our Release Notes here !

To ensure we provide our customers with the best possible experience, we periodically evaluate the usage and effectiveness of existing product features and determine which ones should be deprecated so that we can invest our engineering and support resources on the really valuable and often used features. To make the deprecation process a smooth one, we publish a deprecation schedule on the Socrata Knowledge Base to provide advanced notice and help you plan if youre impacted by the changes.

Feature

Replaced With

Announced On

Deprecation Date*

Notes and Resources

Open Performance

Open Performance Cloud

February 19, 2019

May 1, 2020

The New Performance Solutions Rollout: Open Performance Cloud

Creating reports with legacy Report Builder

Perspectives

August 5, 2019

November 18, 2019

(originally October 1, 2019)

Create a Story

Perspectives Resources

Users can still access and view existing reports created with Report Builder.

opendata.socrata.com

N/A

August 14, 2019

October 31, 2019

Rich List View

N/A

August 26, 2019

September 30, 2019

RSS Exports

N/A

August 26, 2019

December 31, 2019

Deprecation Completed

Feature

Replaced With

Announced On

Deprecated On

Notes and Resources

Legacy Dataset of Datasets

Asset Inventory

November 13, 2018

November 30, 2018

These datasets were replaced by the Asset Inventory and ceased being updated in 2016. They have been deleted due to reliance on deprecated datatypes and lack of utility.

Asset List and Asset Inventory Dataset Overview

Creating Charts with the Classic Visualization Builder

New Visualization Canvas

August 16, 2018

April 8, 2019

Creating a Visualization in the New Visualization Canvas

Product Update: Deprecating Classic Maps and Visualizations

Users can still access and edit existing charts created with the classic chart builder.

Creating Maps with the Classic Map Creation Tool

New Visualization Canvas

August 16, 2018

April 8, 2019

Creating Maps with Point Data

Creating Maps with Lines and Polygons

Creating Multilayer Maps

Product Update: Deprecating Classic Maps and Visualizations

Users can still access and edit maps created with the classic map builder.

Followers

n/a

March 1, 2019

May 13, 2019(originally April 1, 2019)

This is legacy, little-used functionality. It is not being replaced by a new feature.

Socrata Legacy Import (Classic import)

Socrata Data Management Experience (New Import Experience)

August 16, 2018

March 31, 2019 (originally January 7, 2019)

Using the Socrata Data Management Experience

Deprecated Features in the Socrata Data Management Experience

Classic Edit Experience (Append-and-Replace Wizard)

Socrata Data Management Experience (New Edit Experience)

August 16, 2018

March 31, 2019 (originally January 7, 2019)

Editing Datasets with the Socrata Data Management Experience

Creating Calendars with the Classic Tool

n/a

February 1, 2019

August 1, 2019 (originally March 31, 2019)

Routing and Approval and View Moderation

Approvals

November 13, 2018

August 23, 2019 (originally May 31, 2019)

Using New Approvals

How to Approve a Dataset (Video)

Creation of Location Column Types

Point Column Types

February 13, 2019

August 30, 2019

Creating Georeference Columns in the New Publishing Experience

Social sign in

Username-password sign in

July 12, 2019

August 30, 2019

As always, our Global Services team is available to help with any questions you may have about these upcoming transitions. You can contact Socrata Support at [email protected], or you can reach out to your Account Manager for additional information or with any concerns specific to your Socrata site.

*Please note that this is not a product roadmap and is subject to change.

The dataset UID (UniqueIDentifer) is the4x4 of alphanumeric characters at the end of the dataset url.

Every dataset, working copy, chart, map and user account is associated with one.

For example, this is a public dataset on Opendata.socrata.com with a UID of 6v2k-v9qc.

https://opendata.socrata.com/dataset/The-Planes/6v2k-v9qc

For tabular datasets that have an associated Primer page, this URL will direct you to that page. To go directly to the table view,add"/data" to the end of the URL, like this:

https://opendata.socrata.com/dataset/The-Planes/6v2k-v9qc/data

Tip:If you have the unique ID of the asset (dataset, view, visualization) you can access it by using this short URL format:https://DOMAIN/d/4x4

Ex:https://opendata.socrata.com/d/6v2k-v9qc

To sign up for an account on one of our Socrata domains, append "/signup" to the domain URL. This will direct you to a sign up page where you will enter an email address, display name, password, and a reCAPTCHA code.

Your password must be between 8 and 40 characters and satisfy 3 of the following criteria:

contain a digit

contain a lowercase letter

contain an uppercase letter

contain a non alphanumeric symbol (i.e. $%^@!)

Once you have hit Create My Account, an email with a link to activate your account will be sent to the email address you entered.

Note: Sometimes these email notifications get placed in your spam folder, please check your spam folder if you do not receive the email in a timely fashion.

Highlights

When a user creates a Story, they will be provided recommended templates to jump-start their Story creation!

Minor Updates

Catalog & Metadata

Resolved an accessibility issue with disabled options in the Asset Manager dropdown menu.

Data Ingress & Updates

Resolved an issue preventing the ability to search for a specific layer in the Esri connector.

Resolved an issue preventing users from publishing new datasets in IE11.

Improved the error message in DSMP for field name validation errors.

Visualizations

Resolved an issue causing visualizations to render incorrectly in stories when date filters were applied.

Coming Soon!

Expected release - 2 weeks

The creation of reports through Report Builderwill be disabled by November 11, 2019.Reports will remain accessible but read-only.

Expected release - 1 month

Socrata Gateway, a new framework for creating and automating datasets in Socrata by connecting directly to on-premise and cloud based systems, will be available for all customers.

Users will soon have more advanced alert scheduling and threshold triggering options when creating an alert for a dataset.

Administrators will have the ability to configure custom colors and palettes to be used by end-users on a domain-wide basis for creating visualizations.

Legend: *** client-requested functionality

Curious about other areas of the platform? Visit our Performance or SCGC release notes for more! Or give us a shout at [email protected] with questions or ideas.

Technical jargon can be overwhelming, especially acronyms. Everyone at Socrata wants you to be successful in your endeavors to make data accessible, more understandable, and more actionable by everyone who wants to know more about their government and communities. To that end, we've created a glossary of terms that you might hear, or read about, during your journey with Socrata's products. This glossary is meant to be a living document so if you have suggestions for words or acronyms to include, drop us an email note at [email protected] .

To begin using the glossary, click on a letter below to jump to that particular section. If you'd like, you can also scroll through the list to see what you can discover!

A - B - C - D - E - F - G - H - I - J - K - L - M - N - O - P - Q - R - S - T - U - V - W - X - Y - Z

- A -

Action Bar -With the Socrata Data Management Experience also comes a new way to manage your dataset: the action bar. The action bar appears on a dataset's Primer page, the data table, working copies, and revision drafts. This bar replaces the tasks once performed in theManagetab of the data table. It is now easier to create drafts, share with collaborators, and manage your audience.

API - An API (Application Programming Interface), at its most basic level, allows your product or service to talk to other products or services. In this way, an API allows you to open up data and functionality to other developers and to other businesses.

Automate This -A tool in the Socrata Data Management Experience that automatically generates code that you can use to automate updates to your dataset. In the first iteration of this tool, Python code is generated; in subsequent iterations, more languages will be added.

- B -

Bar graph - A bar graph or chart uses horizontal or vertical bars whose lengths proportionally represent values in a dataset. A chart with vertical bars is also called a column graph or chart.

- C -

Catalog - A catalog represents a collection of assets that are grouped into categories. Catalogs organize assets to make it easier to navigate to the information needed.

CSV - CSV (comma separated values) file is a specially formatted, plain text file which stores spreadsheet or basic tabular information in a very simple format, with one record on each line, and each field within that record separated by a comma.

- D -

Data - Facts and statistics collected together for reference or analysis

Dataset - A dataset is an organized collection of data. The most basic representation of a dataset is data elements presented in tabular form. Each column represents a particular variable. Each row corresponds to a given value of that columns variable. A dataset may also present information in a variety of non-tabular formats, such as an extended mark-up language (XML) file, a geospatial data file, or an image file.

DataSync - DataSync is Socratas free, simple, and powerful publishing tool that allows users to schedule and automate their data updates and upload large data files.

DNS - Domain Name System (DNS) The friendly naming system for giving addresses to web servers and webpages.

Data asset - Within a Socrata catalog, assets may include raw, source and federated datasets, as well as visualizations (table, chart, map, data lens, or story).

Data automation - The use of automatic processes, equipment, or systems for the purpose of collecting, processing, storing, transmitting, and presenting of data. Pertains especially to the use of computers and peripheral devices.

Data cleaning - Data cleaning, also called data checking or data validation, is the process b which missing, erroneous, or invalid data are determined and cleaned, or removed, from a dataset and follows the data preparation process.

Data management - Administrative process by which the required data is acquired, validated, stored, protected, and processed, and by which its accessibility, reliability, and timeliness is ensured to satisfy the needs of the data user.

Data preparation -The process by which data are readied for analysis and includes formatting, or normalizing, of values in a dataset.

Data provisioning - The process of making data available in an orderly and secure way to users, application developers, and applications that need it.

Data transformation - The process of converting data or information from one format to another, usually from the format of a source system into the required format of a new destination system.

Data transparency - The ability to easily access and work with data no matter where it is located or what application created it. The assurance that data being reported are accurate and are coming from the official source.

Data type - A classification identifying one of various types of data, such as real, integer or Boolean, that determines the possible values for that type; the operations that can be done on values of that type; the meaning of the data; and the way values of that type can be stored.

Data visualization - The process by which data are visualized, or presented, after the data cleaning process, and involves making choices about which data will be visualized, how data will be visualized, and what message will be shared with the target audience of the visualization. The end result may be referred to as data visualization.

Dependable variable - A type of variable whose value is determined by, or depends on, another variable.

Domain -A group of computers and devices on a network that are administered as a unit with common rules and procedures. Within the Internet, domains are defined by the IP address. All devices sharing a common part of the IP address are said to be in the same domain.

- E -

Egress - Exporting or downloading data

Esri- Environmental Systems Research Institute (Esri) is an international supplier of Geographic Information System (GIS) software, web GIS and geodatabase management applications.

ETL - ETL is short for extract, transform, load, three database functions that are combined into one tool to pull data out of one database and place it into another database.

- F -

Federated dataset - Dataset federation is the ability to have datasets that are published on different Socrata sites or other services brought together into a public Socrata catalog.

Filter - A programmatic way to narrow a search using specified conditions.

Flat file - A file of data that does not contain links to other files or is anon-relational database.

FME - The Feature Manipulation Engine (FME) is a platform that streamlines the translation of spatial data between geometric and digital formats. It is intended especially for use with geographic information system (GIS), computer-aided design (CAD) and raster graphics software. The FME is useful in applications involving interactive geographical, geological and topographical mapping, such as Google Earth and MapQuest. It facilitates the transformation of spatial data into a variety of formats, data models and repositories for transmission to end users. This process is called spatial extract, transform and load (spatial ETL).

Front-end check - A type of data validation for datasets gathered electronically, and is performed at the front end, or before data are stored in an electronic database.

- G -

Geospatial - Used to indicate that data that has a geographic component to it. This means that the records in a dataset have locational information tied to them such as geographic data in the form of coordinates, address, city, or ZIP code. GIS data is a form of geospatial data. Other geospatial data can originate from GPS data, satellite imagery, and geotagging.

GIS - Refers to a system where geographic information is stored in layers and integrated with geographic software programs so that spatial information can be created, stored, manipulated, analyzed, and visualized (mapped).

GTFS - General Transit Feed Specification (GTFS) defines a common format for public transportation schedules and associated geographic information.

- H -

Heat map - A heat map is a graphical representation of data in the form of a map where the individual values contained in a matrix are represented as colors.

- I -

Ingress - Uploading and/or updating data

- J -

JSON -JavaScript Object Notation (JSON) is syntax for storing and exchanging text information. Much like XML.

- K -

KML File - Keyhole Markup Language (KML) is an XML grammar and file format for modeling and storing geographic features such as points, lines, images, polygons, and models for display.

KMZ file - KML files are very often distributed in KMZ files, which are zipped files with a .KMZ extension. When a KMZ file is unzipped, a single doc.kml is found along with any overlay and icon images referenced in the KML and any network-linked KML files.

- L -

None

- M -

Median -A type of average, or measure of central tendency, in which the middle of a dataset is determined by arranging its numeric values in order.

Metadata - Metadata describes a number of characteristics or attributes of data; that is, data that describes data. (ISO 11179-3). For any particular datum, the metadata may describe how the datum is represented, ranges of acceptable values, its label, and its relationship to other data. Metadata also may provide other relevant information, such as the responsible steward, associated laws and regulations, and the access management policy. The metadata for structured data objects describes the structure, data elements, interrelationships, and other characteristics of information, including its creation, disposition, access and handling controls, formats, content, and context, as well as related audit trails.

Mode - a numeric value that appears most often in a dataset.

- N -

None

- O -

OData -Open Data Protocol (OData) is a RESTful data access protocol initially defined by Microsoft. Check out our OData API !

Open Data - Making data that belongs to the public broadly accessible and usable by humans and machines, free of any constraints.

Outlier - An extremely high or extremely low numeric value that lies outside the distribution of most of the values in a dataset.

- P -

PDI - Pentaho Data Integration (PDI, also called Kettle) is the component of Pentaho responsible for the Extract, Transform and Load (ETL) processes. Though ETL tools are most frequently used in data warehouses environments, PDI can also be used for other purposes such as Migrating data between applications or databases, exporting data from databases to flat files, loading data massively into databases, data cleansing and integrating applications.

PDI - Police Data Initiative (PDI) is a White House initiative promoting the use of open data and technology to improve law enforcement practices and outcomes.

Pie Chart - A circular graph divided into sectors, each with an area relative to the whole circle, and is used to represent the frequency of values in a dataset.

Publishing Workflow - The activities and processes that lead to the publication of data, associated metatdata and accompanying documentation.

- Q -

Qualitative data - A type of data that describes the qualities or attributes of something using words or other non-numeric symbols.

Quantitative data - A type of data that quantifies or measures something using numeric values

- R -

RDF - Resource Description Framework(RDF) is a standard model for data interchange on the Web.

Range - A range isdetermined by taking the difference between the highest and lowest numeric values in a dataset.

Raw data - Refers to data that have only been collected, not manipulated or analyzed, from a source.

REST API - The representational state transfer (REST) architectural style describes six constraints: uniform interface, stateless, cacheable, client-server, layered system, and code on demand (optional).

Row Identifier -Socrata datasets are essentially a collection of rows. Each row can be uniquely designated by its row identifier, much like a drivers license number or social security number identifies an individual. For those familiar with database concepts, they essentially act the same way as primary keys.

RSS - With Rich Site Summary (RSS) it is possible to distribute up-to-date Web content from one website to thousands of other websites around the world.

- S -

SaaS - Software as a Service (SaaS), is a software licensing and delivery model in which software is licensed ona subscription basis and is centrally hosted.

SDP - Socrata Data Player (SDP) is a feature of the Socrata Platform which allows audiences to republish data on the web as an Embed. When hosted on webpages, these embeds serve as extensions of the portal - one can interact with the data and, the content updates dynamically when the underlying data is updated.

SEO - Search Engine Optimization (SEO) is the process of getting traffic from the free, organic, editorial or natural listings on search engines, like Google, Yahoo, and Bing.

SSL - Secure Sockets Layer (SSL) is a standard security technology for establishing an encrypted link between a server and a client - typically a server (website) and browser. The link ensures that all data passed between the web server and browsers remain private and integral.

Scatterplot -Uses plotted points (that are not connected by a line) to represent values of a dataset with one or more dependent variables and one independent variable.

Shapefile - A digital vector (non-topological) storage format for storing geometric location and associated attribute information. Shapefiles can support point, line, and area features.

SODA - Socrata Open Data API

Source System - A System of Record (SOR) or Source System of Record (SSOR) is a data management term for an information storage system (commonly implemented on a computer system) that is the authoritative data source for a given data element or piece of information.

Spatial locations - Describes where something (such as a collection) is physically located, using geospatial coordinates such as latitude and longitude.

SQL query - Structured Query Language (SQL) is a special-purpose programming language responsible for querying and editing information stored in a certain database management system.

Stacked bar graph - A type of bar graph whose bars are divided into sub-sections, each of which proportionally represent categories of data in a dataset that can be stacked together to form a larger category.

Standard deviation - A measure of how much the values in a dataset vary, or deviate, from the arithmetic mean by taking the square root of the variance.

Story -The public facing component of the Socrata Tool "Perspectives" used to create data driven narrative

- T -

TSV -TSVstands for Tab Separated Values.TSVfiles are used for raw data and can be imported into and exported from spreadsheet software.TSVfiles are essentially text files, and the raw data can be viewed by text editors, though they are often used when moving raw data between spreadsheets.

- U -

UI - User Interface (UI) is everything designed into an information device with which a human being may interact- including display screen, appearance of a website, help messages, and how an application program or a website invites interaction and responds to it.

- V -

Variance - A measure of how spread out the numeric values in a dataset are, or how much the values vary, from the arithmetic mean.

- W -

None

- X -

XLS - The original file extension used for Microsoft Excel spreadsheets.

XML - XML (extensible markup language) is a simple, very flexible text format derived from SGML (standardized general markup language). XML is Often a Complement to HTML. In many HTML applications, XML is used to store or transport data, while HTML is used to format and display the same data.

- Y -

None

- Z -

None

It is now possible to create new Stories based on new templates designed to meet your needs for Dashboards, Reporting, and more! Templates pre-populate your new Story page with organized content blocks and headers, helping you ensure consistent and professional presentation as you create new Story pages for your domain. Once you choose a template, its content can be interacted with just as flexibly as a normal custom Story layout.

Using Story Templates

1. To get started, create a new Story as usual by clickingCreate in your header bar, selectingStory from the drop-down, and naming your new Story.

2. A generic new Story draft will open. Toward the top right of your Asset Action Bar you will see a templates icon (hovering over this icon will expand to show the "Templates" text). Click this to view available templates:

3. You can now review multiple template options organized into the Landing Page, Dashboards, Reports, and Tracking Measures categories:

4. Select a template to get started:

5. Click into the various content blocks, headers, etc. in the template to complete your Story. Templates are flexible - you can always use one as a jumping-off point and customize as desired.

Socrata Citizen Connect is a highly configurable geospatial interface that connects citizens and elected officials with information about the government services that affect their communities. Citizen Connect links 311, crime, traffic incidents, and other key service information and allows users to explore, analyze, and sign up for alerts around the data.

Citizen Connect allows you to filter by place, date range, and type of data. Customizing the display to best meet your needs. The map is fully interactive and allows users to easily select and view different data points. Through the header bar, you can easily create embeds and share data and set up customized alerts.

Datasets have two permission levels, public and private. Public datasets are accessible to all visitors of your domainand can be discovered through the site-wide data catalog. Private datasets can only be accessed by the dataset owner, users on thesite with the appropriate permissions, or users with whom they've created a share with using the Sharing tool. You can read more about permissions here.

Changing the permissions of the dataset will now be done through the action bar. To do so you will need to select "..." on the bar and thenManage Viewers.

From there you can select the audience of the dataset as Private or Public. The current state of the dataset will be highlighted.

Note: It may take up to 15 minutes for your dataset to be re-indexed and for the site-wide data catalog to be updated. The actual permissions change, however, occurs immediately.

When using the Socrata Data Management Experience, you can now sort your dataset and modify column widths while reviewing or adding data to your dataset.You will first need to create a revision of your dataset using the action bar.

Once in the new tool, you will first want to select eitherAdd DataorReview Data, based on the action you will take.

Primer page

This will load the columns in your dataset. From here you can add sorts to your dataset or adjust column widths.

Sorting

Once your data has loaded you can add a column based sort select the kabob in the right-handcorner of the column. Selecting the kabob menu will launch a modal with a number of options. To sort the dataset based on column values, select one of the sorting options.

Once you have applied your sort, save your changes.

Note*: The sort will not show in the data preview. However, the dataset sort will be applied once the dataset is published.

Adjust Column Widths

While reviewing data, you can also adjust the width of the column which will be applied to the published data table.

To change column widths, select the border between two columns headers and drag the column to adjust its width. The columns can be made more narrow or wide to fit the data in the cell.

Note: Column widths will not be reflected on the table preview, but they will be reflected in the dataset grid view.

Before we can set things up on the Socrata side for Single Sign-On, you will need to set up a trust relationship with Socrata. (Active Directory provides security across multiple domains through interdomain trust relationships). To do this, follow one of these two sets of instructions depending on your identity provider and file a support ticket by emailing [email protected] with the requested information to complete the setup.

ADFS

Please provide this information:

All Socrata-powered sites associated with your organization

Email domain(s) that will be using SSO to sign into your site

ADFS URL or Federation metadata file

SAML

Please provide this information for Support:

All Socrata-powered sites associated with your organization

The name of your Identity Provider (IdP)

E-mail domain(s) that will be using SSO to sign into your site

Sign-in URL

X509 Signing Certificate in PEM or CER format

Sign-out URL (Optional)

User ID Attribute (Optional)

What next?

After we receive the above information from you, we will set up a connection and provide you with the parameters from us that you will need to set up the connection with your Identity Provider. Then, we will arrange a time to begin testing the connection. Typically, we perform testing using a test email domain (i.e. @test.socrata.com). We configure the SSO connection for that email domain and test the login flow using test credentials. Alternatively, if no test email domain is available, we perform testing during a specific hour or two-hour block of time. After a successful testing period, we will enable SSO for your domain using your organizations email domain (i.e. @socrata.com).

You can now view the status of all scheduled updates on a single page! This page will list all updates that are scheduled through the Socrata platform, this currently encompasses Socrata Gateway or datasets that are imported via URL from the Socrata Dataset Management Experience

To navigate to this page, you will start at the administration homepage and choose underData and Assets.

On the schedules page, you can view all scheduled updates, the connection type and status, as well as the timestamps for the last and next update, and the scheduled frequency.

The page also allows you to navigate to tabs for both Gateway and URL to view each component individually.

Formerly the article for Scheduling Data Imported from URL.

With the Socrata Data Management Experience, comes the ability to schedule data updates for datasets that were created by importing from a URL and datasets that are connected through Socrata Gateway.

Watch a short video on this process here: How to Automate Data from a URL

Creating a Scheduled Update

Once a dataset that is created from url or is set up to be updated through Gateway has been published, access the Primer page and select theSchedule Updates option.

For datasets imported by URL this will open a modal where you can configure the frequency of update in days, the time of day the update will occur, a run now option, and the import URL of the source data.

For a Gateway connection, you will have the same options for run now, frequency and time of day, as well as the details for the Agent, Plugin, and data source of the source data. Like datasets imported from url, there will also be a link to change the data source of the Gateway connection.

Editing a Scheduled Update

After scheduling has been setup, you can use the same steps as the above to access and edit it. In addition to the time of day, frequency, and source data, you will also be able to pause/resume or delete an existing schedule.

Updating the Source

To update the URL used, selectchange urlor change datasource settingsfrom the Schedule Automation for URL Datasets modal. This will open a draft of the dataset where a new source can be configured. From there, you can apply additional changes to the dataset such as display formatting, transforms, etc. Once the draft is published, the updated Source of the dataset will be used on the next scheduled run.

Data Publishers can use Socrata Gateway to create and update datasets using the Gateway agent and plugins. Both the Agent and Plugin can be created in the Socrata Dataset Management Experience (SDMX).

Connect to an External Datasource

Provision Agent

Create a Plugin

Creating aDataset

Schedule Dataset Updates

Connect to an External Data Source

The first thing you will do will be to either create a new dataset or edit an existing one - taking you to the main page in SDMX.

You will then need to select Add Datain the Data Actions section.

change directory

From there, you will want to choose the optionConnect to an External Data Source.

You will now be on the Socrata Gateway page for your dataset. This page will list any plugins already created to choose from. If your plugin is already created skip to Connect Datasource. Otherwise, your next step will be to Provision a new agent.

Provision Agent

The first step you will take is to provision a new agent. To do so, click the Provision Agent button at the top right of the page.

This will launch a new window with additional instructions. The first thing you will do is to name your agent and download it. The agent will download as a .zip file containing an executable .jar program and a README* file. Once you agent is downloaded, move on to theNext page.

*Tip: The README file is a great resource with instructions for MacOS, Windows and Linux.

Setup Agent

Once you have downloaded the agent, the next step will be to set it up. The window will allow you to view instructions for Windows, Mac, or Linux machines. In general, the steps to setting up a new agent will be as follows:

Place the downloaded folder on the server or computer you are connecting to Socrata (if its not there already).

Extract/Unzip the downloaded file.

Open the new subfolder containing the zipped contents of the download.

Run the agent as a service

Once the agent is running, go back to the Agent Stepper and click refresh at the bottom of the page, this will prompt Socrata to check to see if the agent is running. You will not be able to move to the next page until the agent is confirmed to be running. Once it is confirmed that the Agent is running, click Next in the window.

Next Steps

You will now have the option to either set up a plugin now or choose to do it a later time, the following section will walk through creating a plugin.

Create a Plugin

A plugin is used to connect directly to a specific source system. Once a plugin is configured, users can create datasets from any data the plugin is configured to access for that source. You can set-up a plugin on the final page of the Agent set up tool or at any point with the Add Plugin button.

When adding a plugin, the first thing you will do is search or browse the list to determine the plugin associated with your source system.

In this example, I have chosen the CSV option. Selecting Set-Up will open a new window with steps to configuring your plugin.

The first page is a plugin overview, it will contain a description and any required fields needed to run the plugin:

On the second page, you will name your plugin, the name should help you understand the data source you are connecting to.

The final page will give you the Set-up instructions for the plugin.

During the agent set up process, if you didnt note the path of your downloaded agent file, gather that path now. It will be called <Gateway_The name you called your agent>.

Open the command prompt (Windows) or Terminal (Mac) and navigate to that folder where you have placed the agent. This can be done using the cd () command.

Copy and paste this command into the command prompt/terminal.

java -jar socrata-ingress-agent.jar --configure-plugin csv:Andrew_CSV_1

Run the command by hitting enter or return. This will download the plugin from Socrata, verify it, and run it on your server.If you get an error, try changing directories.

When the plugin is run it will display a window on your server asking for plugin-specific configuration and credentials (if needed).This information will not live on Socrata.

Fill out the configuration information and click OK.

Once completed you can return to Socrata, the Done option in the setup window should now be available.

Creating a Dataset

Now that your plugin is available, you can select it and connect it to the data source you are using for the dataset.

In this example, I will select the CSV plugin, there is an arrow dropdown that will show a list of all available files that can be published to your dataset. To choose the source data, selectUse this data source.

Selecting the data source will then load the data into Socrata, and you can proceed to set up the dataset just as you would any other!

Schedule Dataset Updates

Once the dataset has been published, access the Primer page and open the dropdown menu to find theSchedule Updatesoption.

This will open a modal where you can configure the frequency of update in days, the time of day the update will occur as well as view or change the datasource settings. In the below example, the dataset would update at 6 AM Pacific time every day:

Once the time and frequency are set, click Save Schedule. The schedule can be modified at any time by choosing the Schedule Updates option.

Congratulations! You now have a fully automated Gateway powered dataset!

Socrata Gateway consists of both Agents and Plugins.The agent is set up on a server that has access to the source system hosting your data. Various plugins exist for your agent to connect to different source systems both on the cloud (Esri, AWS S3, etc) as well as on-premise (MS SQL, Excel, etc).

Provision Agent

Creating Plugin

Creating a Dataset

Schedule Dataset Updates

Managing Agent and Plugin

The first step to setting up your Socrata Gateway will be to navigate to Gatewayfrom the Socrata admin page on your domain. This can also be navigated to directly at https://domain/admin/gateway.

Socrata Dataset Management Experience

Provision Agent

Provision and download agent

Once you are in the Gateway section, you will need to provision a new agent. To do so, click the Provision Agent button at the top right of the page. This will launch a new window with additional instructions. The first thing you will do is to name your agent and download it. The agent will download as a .zip file containing an executable .jar program and a README* file. Once you agent is downloaded, move on to theNext page.

*Tip: The README file is a great resource with instructions for MacOS, Windows and Linux.

Setup Agent

Once you have downloaded the agent, the next step will be to set it up. The window will allow you to view instructions for Windows, Mac, or Linux machines. In general, the steps to setting up a new agent will be as follows:

Place the downloaded folder on the server or computer you are connecting to Socrata (if its not there already).

Extract/Unzip the downloaded file.

Open the new subfolder containing the zipped contents of the download.

Run the agent as a service

Once the agent is running, go back to the Agent Stepper and click refresh at the bottom of the page, this will prompt Socrata to check to see if the agent is running. You will not be able to move to the next page until the agent is confirmed to be running. Once it is confirmed that the Agent is running, click Next in the window.

Next Steps

You will now have the option to either set up a plugin now or choose to do it a later time, the following section will walk through creating a plugin. The Gateway page will also now list your new Agent.

Create a Plugin

A plugin is used to connect directly to a specific source system. Once a plugin is configured, users can create datasets from any data the plugin is configured to access for that source. You can set-up a plugin on the final page of the Agent set up tool within the gateway page or at any point with the Add Plugin button.

Likewise, agents and plugins can be created in the Socrata Dataset Management Experience when creating a new dataset.

When adding a plugin, the first thing you will do is search or browse the list to determine the plugin associated with your source system. In this example, I have chosen the CSV option. Selecting Set-Up will open a new window with steps to configuring your plugin.

The first page is a plugin overview, it will contain a description and any required fields needed to run the plugin:

On the second page, you will name your plugin, the name should help you understand the data source you are connecting to.

The final page will give you the Set-up instructions for the plugin.

During the agent set up process, if you didnt note the path of your downloaded agent file, gather that path now. It will be called <Gateway_The name you called your agent>.

Open the command prompt (Windows) or Terminal (Mac) and navigate to that folder where you have placed the agent. This can be done using the cd ( change directory ) command.

Copy and paste this command into the command prompt/terminal.

java -jar socrata-ingress-agent.jar --configure-plugin csv:Andrew_CSV_1

Run the command by hitting enter or return. This will download the plugin from Socrata, verify it, and run it on your server.If you get an error, try changing directories.

When the plugin is run it will display a window on your server asking for plugin-specific configuration and credentials (if needed).This information will not live on Socrata.

Fill out the configuration information and click OK.

Once completed you can return to Socrata, the Done option in the setup window should now be available. The plugin will also be reflected on the Gateways page:

Creating a Dataset

To create a new dataset that sources data from a plugin, start in the select Add Data > Connect to an External Data Source. This will bring you to a page with a list of available plugins.

If you have not created your agent or plugin yet, you can also do that from this screen following the same instructions as above.

Schedule Dataset Updates

Once the dataset has been published, access the Primer page and open the dropdown menu to find theSchedule Updatesoption.

This will open a modal where you can configure the frequency of update in days, the time of day the update will occur as well as view or change the datasource settings. In the below example, the dataset would update at 6 AM Pacific time every day:

Once the time and frequency are set, click Save Schedule. The schedule can be modified at any time by choosing the Schedule Updates option.

Congratulations! You now have a fully automated Gateway powered dataset!

Manage Agents and Plugins

Agents and Plugins can be managed from the Gateway admin page. At any point, you can Provision a new Agent or add a Plugin. This will show all agents and plugins set up on your domain.

The actions menu also shows two different options: Delete and Copy Snippet. Delete will completely remove the agent, whereas Copy snippet will provide the command used to provision the agent during set up, this can be useful if the agent becomes disconnected. Deleting an agent will pause all scheduled updates for all datasets created from all plugins associated with that agent.

You can also view all plugins connected to a single agent on this page.

Major Updates

Scheduling via UI: Users can now manually run schedules that have been created via the UI rather than waiting until the next scheduled run time.

Minor Updates

Administration & User Management

Users will need to acknowledge the Tyler Privacy Policy and End User Agreement one time upon login.

Catalog & Metadata

Resolved an issue where Data Lens descriptions were rendering with html tags exposed on the asset inventory page.

Data Ingress & Updates

Resolved an issue preventing users from sorting by multiple columns.

Resolved an issue preventing users from adding asset tags by entering a comma separated list of tag values.

Resolved an issue where adding a georeference deleted a datasets first column.

Resolved an issue causing numbers to round incorrectly on primer pages.

Dataset Management & Updates

Resolved an issue where changes weren't saving when editing a working copy.

Users can now mute notifications for individual assets by using the mute button on the primer page.

Visualizations

Added an option to exclude missing or null values when using a text filter.

Coming Soon!

Expected release - 2 weeks

Users will soon have the ability to create a calendar visualization to display basic event information at a monthly view.***

The creation of reports through Report Builder will be disabled by October 31, 2019. Reports will remain accessible but read-only.

Expected release - 1 month

Users will soon have more advanced alert scheduling and threshold triggering options when creating an alert for a dataset.

Administrators will have the ability to configure custom colors and palettes to be used by end-users on a domain-wide basis for creating visualizations.

Legend: *** client-requested functionality

Curious about other areas of the platform? Visit our Performance or SCGC release notes for more! Or give us a shout at [email protected] with questions or ideas.

Whats a catalog?

The catalog is a public facing area of an open data website which allows for browsing, searching, and sorting of publicly available assets. Assets on an open data website may include raw, source and federated datasets, visualizations (table, chart, map, data lens, or story) and derived views.

How can the catalog be used?

Public viewers of your open data site will use the catalog to find the datasets and visualizations they need. Using the catalog, users can perform a keyword search,and filter by a category, view type or topic. Your catalog sorts assets by relevancy (most relevant), access frequency (most accessed), newness to the catalog (recently updated) and most recently updated to allow for efficient and meaningful discovery.

Making the catalog useful

Dataset metadata, such as title and description, play an important role in making your catalog useful to viewers trying to find information on your site. When a viewer performs a catalog search, the search results will display based on a number of factors including metadata, number of views, and frequency of updates. Encouraging data publishers to include thoughtful metadatawith their datasets will ensure that viewers get the most of the catalog.

Newly added or updated publicly available assets will show in your catalog within 1 hour of publishing. Keep in mind that datasets with a private setting will not be visible, or available, in your public catalog. Should you need to change the privacy settings, you can do so within the Manage section of the dataset.

Tips and additional information

Shortcut! At any point in time, if you would like to navigate to the catalog, append "/browse" to the domain URL.

Currently, customizations to the catalog are not possible. If you have requests for improvement or feedback, please contact our Support team by emailing [email protected].

Does your catalog look different than whats shown below? Customers using the older version of the catalog can find information at the Navigating The Dataset Catalog

1. Search:A search can be executed by performing a simple keyword search. Operators, Boolean for example, will not produce results. Results are rendered based on information contained within an assets metadata. The order of search results is determined, in part, by the assets view count and frequency of updates.

2.Categories: Clicking on a category will filter the catalog for all datasets with that category selected in the dataset metadata. These categories are set-up and customized by Administrators through the Admin panel.

View Types: Clicking on one of these will filter the catalog with data of only that type.

Data lens pages: These are the visualization pages that are created from datasets. Clicking on this view type will also display the new visualization

Stories: Pages created based on data, visualizations, etc.

Datasets: These are datasets uploaded into the Socrata platform.

External Datasets: links to datasets stored elsewhere on the web.

Files and Documents: downloadable files such as PDFs or Word documents

Filtered Views: These are saved filtered Datasets. This is a view of the type ofDatasetand will be linked to the original Dataset through the "More Views" tab.

Charts: Charts such as pie charts and bar charts created from Datasets. These will also be linkedto the original Dataset through the "More Views" tab.

Maps: Maps created from geolocation data or geospatial data (shapefile, kml, or kmz file). If it is based on geolocation data, it will also be linkedto the original Dataset through the "More Views" tab.

Calendars: Calendars created from Date&Time data in a Dataset.These will be linkedto the original Dataset through the "More Views" tab.

Forms: Forms are created from a Dataset, to collect information from users.These will be linkedto the original Dataset through the "More Views" tab.

Topics: Clicking on a Topic will filter the catalog by all datasets tagged with that word. This section is populated with terms as they are tagged on Datasets in the metadata.

3. Sort: You can sort the catalog in a number of different ways:

Alphabetical: Sorted alphanumerically.

Most Relevant: This is generally associated with a search term, and is calculated based on a normalization of the number of views.

Most Accessed:This is sorted by the number of views each returned result has.

Recently Updated: Sorted by the most recently updated datasets. This also takes into account newly added datasets, as their most recent update is the date of upload.