scan_string() return token_type::parse_error; when parse ansi file

[sdhongjun]

With VS2015 IDE when I run below demo code to read json file, scan_string function return token_type::parse_error at line 2186.
{
ofstream out_json("C:\test.json");

    json jsDefault = json();
    jsDefault["name"] = "默认";
    jsDefault["param"] = json();
    json jsArray = json::array({ jsDefault });

    json jsObj = json();
    jsObj["select"] = "默认";
    jsObj["items"] = jsArray;

    out_json << std::setw(4) << jsObj;

    out_json.close();

    ifstream in_json("C:\\test.json");
    json jsNewObj = json();
    in_json >> jsNewObj;
}

[nlohmann]

Could you please try the develop version and paste the full exception message? On MSVC, we had several issues regarding the encoding of the strings: the library only supports UTF-8 encoded strings.

[sdhongjun]

@nlohmann I'm already using the develop branch, the scan_string function parse error with return token_type::parse_error.

{
    if (JSON_UNLIKELY(not next_byte_in_range({0x80, 0xBF})))
    {
        return token_type::parse_error;//line 2186
    }
    break;
}

When I comment line from 2153 to 2266 and modify default branch to below it works fine. By the way are you have utf-8 convert function.

default:
{
    add(current);
    break;
    /* error_message = "invalid string: ill-formed UTF-8 byte";
    return token_type::parse_error;*/
}

[gregmarr]

Looks like you need to convert your non-ascii strings jsDefault["name"] and jsObj["select"] to utf-8 before you put them in the json object.

[sdhongjun]

@gregmarr Yes! But this project should support this.

[nlohmann]

The project supports UTF-8. Could you try adding a u8 prefix to your string literals, e.g. jsDefault["name"] = u8"默认";?

[sdhongjun]

@nlohmann add u8 prefix not work correct.

#include "json.hpp"

#include <fstream>

using namespace std;
using json = nlohmann::json;

int main()
{
    ofstream out_json("C:\\test.json");

    json jsDefault = json();
    jsDefault["name"] = u8"默认";
    jsDefault["param"] = json();
    json jsArray = json::array({ jsDefault });

    json jsObj = json();
    jsObj["select"] = u8"默认";
    jsObj["items"] = jsArray;

    out_json << std::setw(4) << jsObj;

    out_json.close();

    ifstream in_json("C:\\test.json");
    json jsNewObj = json();
    in_json >> jsNewObj;
    string strJson = jsNewObj.dump(4);

    return 0;
}

Output file c:\test.json text.

{
    "items": [
        {
            "name": "默认",
            "param": null
        }
    ],
    "select": "默认"
}

And string strJson = jsNewObj.dump(4); text is:

{
    "items": [
        {
            "name": "榛樿",
            "param": null
        }
    ],
    "select": "榛樿"
}

[nlohmann]

Can you please attach your code and the JSON file (best as a ZIP archive) so I can check this myself?

[sdhongjun]

Here is VS2015 test project you can try it.
TestJson.zip

[nlohmann]

I don't have MSVC. Can you please execute the code on your side and attach the generated JSON file please?

[sdhongjun]

Check the attach file.

test.zip

[nlohmann]

I can parse the file without problems:

#include <iostream>
#include <fstream>
#include "json.hpp"

using json = nlohmann::json;

int main(int argc, char* argv[]) {
    std::ifstream f("test.json");
    json j;
    f >> j;
    std::cout << j << std::endl;
}

[sdhongjun]

Not error,decode string is not same as src string.

[nlohmann]

In #812 (comment) you mentioned a parse error. I cannot reproduce this with the file. I think your input is not UTF-8 encoded. The library only supports UTF-8.