sbs and hc

CHANGELOG.md

@@ -15,6 +15,17 @@
   - Sends the previous user's open view duration and unsaved custom property changes under the previous ID before switching
   - Ignores remote config values that were still downloading for the previous device ID. The "fetch_remote_config" callback then gets an error
 - ! Minor breaking change ! In a cluster, a worker now asks the main process once whether it runs the SDK, and the main process answers. These messages have a "cly" property, and your own "message" handlers in the main process and in workers receive them as well
+- ! Minor breaking change ! Added SDK behavior settings. The SDK now fetches the settings set in the dashboard's SDK Manager at start, every 4 hours (or at the refresh interval set there) and after a device ID change, and follows them:
+  - The settings can turn off all tracking, all sending, or sessions, views, custom events, crashes and location one by one, change the queue, batch and length limits, and turn the consent requirement on or off, also one set with "require_consent", as on Android and Web. Without that setting the "require_consent" value applies
+  - A setting set in SDK Manager takes precedence over the matching init option. A setting removed there goes back to the init option, or else to the default
+  - The fetched settings are stored and used from the next start on, also before the server answers. The new init option "behavior_settings" provides settings, in the format SDK Manager exports, for the time before the first successful fetch
+  - While tracking is turned off nothing new is recorded, including consent changes and "request" calls, and data recorded before is still sent. While sending is turned off data is kept and sent once sending is turned on again, "flush" resolves false right away and "fetch_remote_config" calls its callback with an error
+  - The settings request is sent also when consent is required and not given, and while sending is turned off, so that changes reach the SDK. Set the new init option "disable_sdk_behavior_settings_updates" to not fetch the settings; stored and provided settings still apply
+  - After a device ID change with merge, the settings are fetched once the merge request is delivered
+- ! Minor breaking change ! Added a health check. Once per start, after the settings fetch, the SDK sends the number of errors and warnings it logged (also when logging is off), the status code and the first 1000 characters of the answer of the last request the server refused, and how often sending paused, together with the app version. The numbers are kept until the server confirms them. The health check is sent also when consent is required and not given, and not while sending is turned off. Set the new init option "disable_health_check" to not send it
+- ! Minor breaking change ! After a request the server answered slowly (in 10 seconds or more by default), the SDK now pauses sending for 60 seconds when the queue is at most half full and the request was recorded within the last 24 hours, so that a busy server can recover. SDK Manager can change these values, and "flush" does not wait for the pause. Set the new init option "disable_backoff_mechanism" to never pause
+- SDK Manager can now set a maximum age for queued requests. Older requests are then dropped instead of sent, and a "flush" call waiting for them resolves false. This is off by default
+- A missing storage file, which is normal before the SDK first writes it, is no longer logged as a warning
 - The npm package now contains only the SDK code, README, LICENSE and CHANGELOG. Tests, examples and repository configuration files are no longer included
 - Added a new method "flush" that sends all recorded data right away and tells when it is delivered, for example before a script ends or in a shutdown handler before "process.exit()"
 - Added a new init time flag "standalone_worker" that lets a cluster worker send its own data when the main process does not run the SDK, for example in PM2 cluster mode:
@@ -27,6 +38,7 @@
 - Fixed a bug where a request was removed from the stored queue before the server confirmed it, so it was lost if the process ended meanwhile
 - Fixed a bug where creating a CountlyBulk instance changed where, or whether, the SDK and other CountlyBulk instances stored their data, and where two CountlyBulk instances could send each other's data under the wrong app
 - Fixed a bug where removing consent was never reported to the server
+- Fixed a bug where a consent change made less than a second before a device ID change was reported under the new device ID or not at all, and where a consent change made right before "flush" and "process.exit()" was lost. Such a change is now sent right away, under the device ID of the user who made it
 - Fixed a bug where consent changes of CountlyBulk users were never sent
 - Fixed a bug where a BigInt value or an object that refers to itself in event, view, user property or crash data crashed the app. BigInt values are now sent as text and self references are left out
 - Fixed a bug where "change_id" with an empty value stopped all tracking. Such values are now ignored with a warning, and an empty device ID stored by an older version is ignored at start
@@ -35,6 +47,7 @@
 - Fixed a bug where "request" cut every value to 256 characters, which broke raw event and user payloads. Raw requests now also count towards the request queue size
 - Fixed a bug where requests containing an apostrophe failed the checksum check of Countly 26.01 and newer servers when "salt" is set. The characters ! ' ( ) * are now percent-encoded
 - Fixed a bug where a value containing half of an emoji, for example a device ID cut in the middle of an emoji, could never be sent and stopped all further sending, also after a restart. Such half characters are now sent as the replacement character "�"
+- Fixed a bug where the key length and segmentation limits also cut Countly's own events, so the server no longer recognized them. For example, with a key length limit of 8 a view was sent as "[CLY]_vi". Views, star ratings, feedback and other internal events now keep their key and segmentation; the segmentation given to "track_view" is still limited
 - Fixed a bug where a CountlyBulk user created with "ip_address" threw an error on most calls
 - Fixed a bug where giving consent to a CountlyBulk user after a "begin_session" or "report_view" call that waited for consent crashed the app. The waiting call is now sent
 - Fixed a bug where the CountlyBulk "start" callback reported an empty queue while data was still waiting for a retry or being sent

lib/countly-behavior.js

@@ -0,0 +1,222 @@
+/**
+ * SDK behavior settings: checks the settings the server sends (from SDK Manager) or the developer provides,
+ * and decides the value of each setting the SDK uses.
+ * @module lib/countly-behavior
+ */
+
+/**
+ * Settings the Node.js SDK follows, with the kind of value each one takes:
+ * "switch" true or false; "count" a whole number of at least 1; "zeroOrMore" a whole number of at least 0;
+ * "fraction" a number above 0 and at most 1
+ */
+var SUPPORTED = {
+    tracking: "switch",
+    networking: "switch",
+    st: "switch",
+    vt: "switch",
+    cet: "switch",
+    crt: "switch",
+    lt: "switch",
+    cr: "switch",
+    bom: "switch",
+    rqs: "count",
+    eqs: "count",
+    sui: "count",
+    lkl: "count",
+    lvs: "count",
+    lsv: "count",
+    lbc: "count",
+    ltlpt: "count",
+    ltl: "count",
+    scui: "count",
+    bom_at: "count",
+    bom_ra: "count",
+    bom_d: "count",
+    dort: "zeroOrMore",
+    bom_rqp: "fraction",
+};
+
+/**
+ * Values used when neither the server nor the developer gives one
+ */
+var DEFAULTS = {
+    tracking: true,
+    networking: true,
+    st: true,
+    vt: true,
+    cet: true,
+    crt: true,
+    lt: true,
+    cr: false,
+    bom: true,
+    scui: 4,
+    dort: 0,
+    bom_at: 10,
+    bom_rqp: 0.5,
+    bom_ra: 24,
+    bom_d: 60,
+};
+
+/**
+ * Longest settings refresh interval in hours. A Node.js timer cannot wait longer than about 24.8 days, and a
+ * longer value would make it fire at once, over and over.
+ */
+var MAX_REFRESH_HOURS = 596;
+
+/**
+ * Tells whether a value is a plain object (not null, not an array)
+ * @param {*} value - value to check
+ * @returns {Boolean} true for a plain object
+ */
+function isPlainObject(value) {
+    return Object.prototype.toString.call(value) === "[object Object]";
+}
+
+/**
+ * Tells whether a value fits the kind a setting takes
+ * @param {String} kind - "switch", "count", "zeroOrMore" or "fraction"
+ * @param {*} value - value from the server or the developer
+ * @returns {Boolean} true if the value can be used
+ */
+function isValidValue(kind, value) {
+    if (kind === "switch") {
+        return typeof value === "boolean";
+    }
+    if (typeof value !== "number" || !Number.isFinite(value)) {
+        return false;
+    }
+    if (kind === "count") {
+        return Math.floor(value) === value && value >= 1;
+    }
+    if (kind === "zeroOrMore") {
+        return Math.floor(value) === value && value >= 0;
+    }
+    return value > 0 && value <= 1;
+}
+
+/**
+ * Keeps only the supported settings with a usable value. Other keys (for example the dashboard's
+ * "bom_preset", or settings for other platforms) are left out, and an unusable value is left out with a
+ * warning, so the developer value or the default stays in use for it.
+ * @param {Object} config - the "c" object of a settings answer
+ * @param {Function} warn - called with a message for each left out value
+ * @returns {Object} the usable settings
+ */
+function normalize(config, warn) {
+    var result = {};
+    Object.keys(config).forEach((key) => {
+        if (!Object.prototype.hasOwnProperty.call(SUPPORTED, key)) {
+            return;
+        }
+        if (isValidValue(SUPPORTED[key], config[key])) {
+            result[key] = key === "scui" ? Math.min(config[key], MAX_REFRESH_HOURS) : config[key];
+        }
+        else if (typeof warn === "function") {
+            warn(`The behavior setting [${key}] has an unusable value [${JSON.stringify(config[key])}], it is ignored.`);
+        }
+    });
+    return result;
+}
+
+/**
+ * Reads the server's answer to a settings request. The answer must be a JSON object with a "c" object;
+ * other top-level keys (v, t, lg, ct and future ones) are ignored, so both the v1 and the v2 format work.
+ * @param {String} responseText - the answer body
+ * @param {Function} warn - called with a message for each problem
+ * @returns {Object|null} the usable settings, or null when the answer cannot be used
+ */
+function parseAnswer(responseText, warn) {
+    var answer;
+    try {
+        answer = JSON.parse(responseText);
+    }
+    catch (ex) {
+        return null;
+    }
+    if (!isPlainObject(answer) || !isPlainObject(answer.c)) {
+        return null;
+    }
+    return normalize(answer.c, warn);
+}
+
+/**
+ * Reads the settings a developer provides with the "behavior_settings" init option, in the same format the
+ * server sends: an object with a "c" object
+ * @param {*} provided - the init option value
+ * @param {Function} warn - called with a message for each problem
+ * @returns {Object|null} the usable settings, or null when none are provided or the format is wrong
+ */
+function parseProvided(provided, warn) {
+    if (typeof provided === "undefined" || provided === null) {
+        return null;
+    }
+    if (!isPlainObject(provided) || !isPlainObject(provided.c)) {
+        if (typeof warn === "function") {
+            warn("behavior_settings must be an object with a \"c\" object, as SDK Manager exports it. It is ignored.");
+        }
+        return null;
+    }
+    return normalize(provided.c, warn);
+}
+
+/**
+ * Reads settings stored by an earlier run
+ * @param {*} stored - the stored value
+ * @returns {Object|null} the usable settings, or null when nothing usable is stored
+ */
+function parseStored(stored) {
+    if (!isPlainObject(stored)) {
+        return null;
+    }
+    return normalize(stored);
+}
+
+/**
+ * Decides the value of a setting: the server's (or the stored or provided) value when there is one, else
+ * the developer's value, else the default
+ * @param {Object} settings - settings in use
+ * @param {String} key - setting key
+ * @param {*=} developerValue - value set with an init option, undefined when none
+ * @returns {*} the value to use
+ */
+function valueOf(settings, key, developerValue) {
+    if (settings && Object.prototype.hasOwnProperty.call(settings, key)) {
+        return settings[key];
+    }
+    if (typeof developerValue !== "undefined") {
+        return developerValue;
+    }
+    return DEFAULTS[key];
+}
+
+/**
+ * Decides whether sending must pause after a successful request, following the backoff settings: the
+ * request took at least bom_at seconds, the queue is at most bom_rqp of its size, and the request was
+ * recorded at most bom_ra hours ago
+ * @param {Object} input - measurements and settings
+ * @param {Number} input.durationSeconds - how long the request took
+ * @param {Number} input.queueLength - requests left in the queue
+ * @param {Number} input.queueSize - maximum queue size
+ * @param {Number} input.ageHours - hours since the request was recorded
+ * @param {Number} input.bom_at - response time threshold in seconds
+ * @param {Number} input.bom_rqp - queue fill threshold as a fraction of the queue size
+ * @param {Number} input.bom_ra - request age threshold in hours
+ * @returns {Boolean} true when sending must pause
+ */
+function shouldBackOff(input) {
+    return input.durationSeconds >= input.bom_at
+        && input.queueLength <= input.queueSize * input.bom_rqp
+        && input.ageHours <= input.bom_ra;
+}
+
+module.exports = {
+    SUPPORTED,
+    DEFAULTS,
+    MAX_REFRESH_HOURS,
+    normalize,
+    parseAnswer,
+    parseProvided,
+    parseStored,
+    valueOf,
+    shouldBackOff,
+};

lib/countly-common.js

@@ -62,6 +62,8 @@ var cc = {
     debugBulk: false,
     debugBulkUser: false,
     warnedOnce: {},
+    // called with the level of every error and warning log line, also when logging is off; the SDK counts them for its health check
+    onErrorOrWarning: null,
     /**
      * Prints a warning once per process, also when logging is off. Used only for setups in which the SDK
      * cannot send or keep data at all, so integrators learn about them without turning on debug logs.
@@ -326,6 +328,9 @@ var cc = {
      * @param {string} message - any string message
      */
     log: function log(level, message, ...args) {
+        if (typeof this.onErrorOrWarning === "function" && (level === this.logLevelEnums.ERROR || level === this.logLevelEnums.WARNING)) {
+            this.onErrorOrWarning(level);
+        }
         if ((this.debug || this.debugBulk || this.debugBulkUser) && typeof console !== "undefined") {
             if (args[0] instanceof Error) {
                 args[0] = args[0].stack || String(args[0]);

lib/countly-storage.js

@@ -440,7 +440,8 @@ function createStorage() {
     }
 
     /**
-     *  Read value from file
+     *  Read value from file. A missing file is normal before the first write, so only other read errors are
+     *  logged as warnings (the health check counts warnings).
      *  @param {String} key - key for file
      *  @returns {varies} value in file
      */
@@ -451,7 +452,12 @@ function createStorage() {
             data = fs.readFileSync(file, 'utf8');
         }
         catch (ex) {
-            cc.log(cc.logLevelEnums.WARNING, `readFile, File not found for key: [${key}]. Returning null.`);
+            if (ex && ex.code === "ENOENT") {
+                cc.log(cc.logLevelEnums.DEBUG, `readFile, File not found for key: [${key}]. Returning null.`);
+            }
+            else {
+                cc.log(cc.logLevelEnums.WARNING, `readFile, Could not read the file for key: [${key}]. Error: [${ex}]. Returning null.`);
+            }
             return null;
         }
 
@@ -761,6 +767,7 @@ module.exports = {
     storeRemove: defaultStorage.storeRemove,
     writeFile: defaultStorage.writeFile,
     forceStore: defaultStorage.forceStore,
+    flushSync: defaultStorage.flushSync,
     getStoragePath: defaultStorage.getStoragePath,
     setStoragePath: defaultStorage.setStoragePath,
     resetStorage: defaultStorage.resetStorage,