Data Import

Viafoura supports importing your existing user and comment data into the audience engagement platform. Viafoura stores its comment data in a unique way that is optimized for quick access and data resiliency. To facilitate this, clients will be required to provide their data in two standardized JavaScript Object Notation (JSON) formats, which are described in this document. Our system then parses this data and inserts it into your Viafoura instance.

The two components of data required for import are users, containers, comments, and likes. Users are fairly straightforward. Containers are the entities where comments relating to the same topic reside, such as comments on a webpage, and contain nested comments. Likes (and dislikes) are user reactions to comments.

Getting Started

After reading this document in its entirety, please provide us sample files for approximately 10,000 comments (in containers) and associated users for a test import. We will load this data into a test environment for us to validate the data format. After verification, we will provide you with additional instructions to upload the full data set to be imported.

JSON Format Requirements

All files must be valid JSON formatted.

Two types of files must be supplied:
_User Data File. Defines all users referenced in the following file as an array of users.
_Container, Comments, and Likes Data File. Defines containers that hold comments, the comments, and lists users who have liked the comments. Likes are optional.

It is recommended, due to the anticipated size of files, to not “pretty format” these files as this will significantly increase file sizes.

Standard JSON formatting rules apply. For example, to have a multi-line string value, you would need to use “\n” as a new-line character.

If a mandatory field is missing in a record, the record will not be imported.

Optional data items can be omitted from the JSON file; do not add optional items as : null, remove the key instead as null is not as same as not providing an optional value. But please keep in mind any comment associated with such items below, as a defaulted value might be supplied by our import process.

HTML tags, for example “
” or “<é” will be imported as-is; in other words, no translation will be done. This requirement simplifies using our API for non-web purposes, such as in mobile applications or when sending data in an email.
It is highly recommended that HTML tags are removed before data is written to JSON files. If not these tags will show as text instead of providing text formatting.

Date-Times must conform to ISO-8601 formats for a timestamp consisting of a date and time.
Examples:

• 2019-12-03T10:15:30
• 2019-12-03T10:15:30+01:00
• 2019-12-03T10:15:30+01:00[Europe/Paris]

Timestamps without an offset will be treated as UTC values, while offsets are treated as offsets relative to UTC.

Organizing Your Data

Large files (over 1GB) may be split into multiple files, though each file must be a complete file satisfying the requirements laid out in this document (you cannot ‘break’ file into two at an arbitrary position as the second file would be missing parent information).

Each file must be valid JSON on its own.

The user data files are the first files to be imported and have no dependencies on other files. Container, comments, and likes files will refer to users in these files.

Comments data should contain linked user data. The referring fields to a user must already be supplied earlier in the import files.

Users at Viafoura

Site groups and Syndication

Sites for a client of Viafoura's may be organized into one or more site groups. A site group allows users and content to be shared between multiple sites, enabling content syndication. If you wish to enable syndication across a site group in the future, the users you provide us must be the same across all those sites. In particular, the "sub" field for a user (described in detail later) must be the same across the sites.

The users for a site group can be supplied in multiple files, but they should all represent the same pool of users.

It is also possible to have multiple site groups if that better suits your needs, and in that case, please group your Users files appropriately.

If you have any questions, please contact Viafoura support.

Social Providers and Authentication

Viafoura supports integration with many external authentication sources. Broadly, authentication sources can be broken down into three categories:

• First Party users: These are users who exist only inside the Viafoura system, without an external login mechanism. These users log in using an email address.
• Second Party users: These users exist at the authentication provider. This may be an authentication provider that you host, or a login vendor such as LoginRadius, Gigya, or Janrain.
• Third Party users: These users log in via an authentication provider, but ultimately identify themselves by logging in elsewhere. An example would be a Google or Facebook user logging in via LoginRadius or Gigya.

It is possible to have a mix of all three users to import.

If you wish to import First Party users, then only an email address must be provided. We do not support importing passwords, and the users must request a password reset email when logging in for the first time after migration. You should communicate this to your users.

For Second Party users, we currently support LoginRadius, Gigya, Janrain, and custom cookie logins. Custom cookie logins receive a token on your webpage, pass it to our servers, then our servers pass it to your servers for validation. If your server accepts the token, it must return a JSON with fields describing each of a unique user id (sub), a display name (name), and optionally an email address. Please contact support for more information regarding cookie login.

For Third Party users, we support Apple, Facebook, Google, LinkedIn, Microsoft, Twitter, and Yahoo.

OpenID Connect (OIDC)

Viafoura supports OpenID Connect integration. Viafoura needs the well-known url of the ID provider as well as your client id. After these two are set up on Viafoura’s admin panel, we are able to accept token IDs and use them for authentication.

OpenID Connect Federation

Viafoura may implement OpenID Connection Federation once the specification is out of the draft stage.

User Data Format

Our user import format is based on OpenID Connect standard claims format.

If there are any errors, missing fields, or inconsistencies in the supplied JSON, the records will not be imported.

Example of User Data file

Filename: users.json
(pretty-formatted for clarity only)

{
   "users":[
      {
         "sub":"daffy_duck",
         "name":"Daffy Duck",
         "email":"[email protected]",
         "email_verified":true,
         "created_at":"2020-10-03T10:15:30",
         "updated_at":"2020-11-09T10:15:30",
         "originating_provider":"email",
         "originating_provider_id":"[email protected]",
         "ban_type":"none"
      },
      {
         "sub":"clark kent",
         "name":"Clark Kent",
         "email":"[email protected]",
         "email_verified":false,
         "created_at":"2020-10-03T10:16:30",
         "updated_at":"2020-11-09T10:19:35",
   "originating_provider":"facebook",
         "originating_provider_id":"123456789ABC",
         "ban_type":"no_posting",
         "ban_expiry":"2021-10-03T10:16:30"
      },
      {
         "sub":"freddy_104",
         "name":"Fred Van der Vleet",
         "created_at":"2020-10-13T11:16:30",
         "updated_at":"2020-10-13T11:16:30",
	   "originating_provider":"cookie",
         "originating_provider_id":"freddy_104"
      },
      {
         "sub":"newbie",
         "name":"Larry Cableguy",
         "created_at":"2020-10-13T11:17:30",
         "updated_at":"2020-10-13T11:17:30",
	   "originating_provider":"oidc",
         "originating_provider_id":"my_oidc_id"
      }

   ]
}

Containers, Comments and Likes Data Format

Likes are nested in comments, and comments are nested in containers. As such, each object type will be described separately.

If there are any errors, missing fields, or inconsistencies in the supplied JSON, the records will not be imported.

Containers Data Format

Comments Data Format

Likes Data Format

Note that it is invalid to provide more than one like object with the same sub for a comment.

Example of Containers, Comments and Likes Data File

Filename: comments.json
(pretty-formatted for clarity only)

{
  "containers": [
    {
      "id": "12345",
      "": "https://example.com/comics/batman/batman.html",
      "title": "This is the \"fantastic\" Batman",
      "description": "The fantastic Batman",
      "created_at": "2020-10-13T11:17:30",
      "updated_at": "2020-10-13T11:18:30",
      "comments": [
        {
          "id": "45678",
          "sub": "daffy_duck",
          "content": "I think it is important that all ducks quack and repel water",
          "created_at": "2020-10-13T11:16:30",
          "updated_at": "2020-10-14T11:16:30",
          "status": "enabled",
          "likes": [],
          "comments": [
            {
              "id": "4343",
              "sub": "mrsmith",
              "content": "totally agreed!",
              "created_at": "2020-11-13T11:16:30",
              "updated_at": "2020-11-14T11:16:30",
              "status": "enabled",
              "likes": [
                {
                  "sub": "sherlock_holmes",
                  "status": "like",
                  "created_at": "2020-10-23T10:16:30",
                  "updated_at": "2020-10-23T10:16:55"
                }
              ],
              "comments": []
            }
          ]
        },
        {
          "id": "45943",
          "sub": "clark kent",
          "content": "If you are superman, you can both quack, repel water and fly",
          "created_at": "2020-10-15T11:16:31",
          "updated_at": "2020-10-15T11:17:31",
          "status": "enabled",
          "likes": [
            {
              "sub": "freddy_104",
              "status": "dislike",
              "created_at": "2020-10-23T10:16:32",
              "updated_at": "2020-10-23T10:16:32"
            },
            {
              "sub": "sherlock_holmes",
              "status": "like",
              "created_at": "2020-10-25T10:17:30",
              "updated_at": "2020-10-25T10:17:30"
            }
          ],
          "comments": []
        }
      ]
    },
    {
      "id": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d",
      "sub": "clark kent",
      "path": "https://example.com/comics/robin/robin.html",
      "title": "I love Batman AND Robin",
      "created_at": "2020-10-13T11:16:35",
      "updated_at": "2020-10-13T11:16:40",
      "likes": [],
      "comments": [
        {
          "id": "0000eb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d",
          "sub": "bob the third",
          "content": "I call \"fowl\" on this stupid discussion",
          "created_at": "2020-10-15T11:17:31",
          "updated_at": "2020-10-15T11:17:31",
          "status": "spam",
          "likes": [],
          "comments": []
        }
      ]
    }
  ]
}

Client-Side Validation Instructions For User Data

The client-side validation process may not provide 100% accuracy, but it drastically reduces the time & manual labour required to resolve the errors due to incorrectly crafted data files for our clients and ourselves. Please follow the following steps to set up your environment for client-side validation.

• Download & install node & npm
• Install the following libraries:
  ajv-cli library -> npm install -g ajv-cli
  ajv-keywords library -> npm install -g ajv-keywords
  *ajv-formats library -> npm install -g ajv-formats
  Download user schema
  _It is suggested to initially run this validation on a small subset of data first to ensure that the data format written to the users.json file is in the correct format. Note that this validation only validates data formats and does not perform any validation relating to data relationships between different fields.
  _Run the validation against the user schema and user data (the following commands assume that you have the schema file user_schema.json and client’s user data file ‘users.json` located inside the current directory you’re running the commands at).

Commands to run validation:

ajv -s user_schema.json -d users.json --strict=false -c ajv-formats -c ajv-keywords --all-errors --errors=json > output.json 2>&1

ajv -s user_schema.json -d users.json --strict=false -c ajv-formats -c ajv-keywords --verbose --all-errors --errors=json > output.json 2>&1

If the above commands seem to hang, try removing the “-c ajv-keywords” and re-run.

If the above command completed with a message like this:
error: Cannot create a string longer than 0x1fffffe8 characters
The file to validate is too big;
the validator cannot be used to verify this file.

Check the produced output.json file for errors.

Client-Side Validation Instructions For Comment Data

As mentioned in the section “Client-Side Validation Instructions For User Data”, download and install node & npm, ajv-cli, ajv-keywords, ajv-formats.

Download comments/likes schema

Run the validation against the comment schema and use data (the following commands assume that you’ve the schema file comment_schema.json and the comment data file ‘comments.json` located inside the current directory you’re running the commands at):

ajv -s comment_schema.json -d comments.json --strict=false -c ajv-formats -c ajv-keywords --all-errors --errors=json > output.json 2>&1

ajv -s comment_schema.json -d comments.json --strict=false -c ajv-formats -c ajv-keywords --verbose --all-errors --errors=json > output.json 2>&1

If the above commands seem to hang, try removing the “-c ajv-keywords” and re-run.

Check the produced output.json file for errors.

Known differences between client-side and server-side validator

Users:

While the server-side validator uses Apache Commons’ email validator that goes beyond the RFC 5322 addr-spec syntax, client-side validator uses RFC 5322. So emails like [email protected] or [email protected] would be evaluated as valid emails for the client-side validator, while they would be invalid for the server-side validator. These users would be ignored by the importer during the actual import process.

Certain value comparisons for dates would only be validated by the server-side validator, for example; updated_at should be later than created_at, ban_expiry should be later than created_at. These cases would be reported at the dry-run that we perform before the actual run. You would need to fix these issues and resend the updated files.

Containers/Comments/Likes:

We suggest you initially run this validation on a small subset of data first to ensure that the data format written to the comments.json file is in the correct format. Note that this validation only validates data formats and does not perform any validation relating to data relationships between different fields and between users.json and comments.json..

Value comparisons for dates as mentioned above also apply here.

There are no semantic checks for the content property of the comments to identify whether html tags are used or not.

Common Errors When Importing Data

• The originating_provider values should be submitted according to their description. Providing incorrect values delay the process, and may require you to modify & resubmit the files.
• Optional values shouldn’t be set to null, please remove them from the file instead. This would semantically fix the problem, but also reduce the size of the import file.
  • Character length constraints should be followed. For example, a user name’s length should not exceed 250 characters.
• Name field can be set to a value like Not Provided if it’s not available. Specifying null or empty strings won’t be enough.
• The updated_at dates should come after the created_at dates.
• If the email field is provided, this email_verified should be provided as well. The email_verified property shouldn’t be provided if email property is not specified.
• Content value for the comments should not include html tags.
• Specifying the same container with the same id but different set of comments in different places within the import file causes the importer to use update mode instead of the insert mode. This is a recoverable behavior for us but it complicates the import file, reduces performance of the importer and increases the import file size due to duplicate values. Merging these records into one entry per the same container would fix all of these issues at once.
• A message indicating that the file is too big means the validator cannot use this file. We recommend keeping your files under 100MB each.