From owner-svn-src-all@freebsd.org Sun Apr 17 21:30:42 2016 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 65D01B12C95; Sun, 17 Apr 2016 21:30:42 +0000 (UTC) (envelope-from bapt@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F1CD31FD9; Sun, 17 Apr 2016 21:30:41 +0000 (UTC) (envelope-from bapt@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id u3HLUfcQ076271; Sun, 17 Apr 2016 21:30:41 GMT (envelope-from bapt@FreeBSD.org) Received: (from bapt@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id u3HLUe8u076262; Sun, 17 Apr 2016 21:30:40 GMT (envelope-from bapt@FreeBSD.org) Message-Id: <201604172130.u3HLUe8u076262@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bapt set sender to bapt@FreeBSD.org using -f From: Baptiste Daroussin Date: Sun, 17 Apr 2016 21:30:40 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r298166 - in head/contrib/libucl: . include lua python python/src src tests tests/basic tests/schema utils X-SVN-Group: head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Apr 2016 21:30:42 -0000 Author: bapt Date: Sun Apr 17 21:30:40 2016 New Revision: 298166 URL: https://svnweb.freebsd.org/changeset/base/298166 Log: Import libucl 0.8.0 Added: head/contrib/libucl/tests/basic/escapes.in - copied unchanged from r298163, vendor/libucl/dist/tests/basic/escapes.in head/contrib/libucl/tests/basic/escapes.res - copied unchanged from r298163, vendor/libucl/dist/tests/basic/escapes.res head/contrib/libucl/tests/basic/load.in - copied unchanged from r298163, vendor/libucl/dist/tests/basic/load.in head/contrib/libucl/tests/basic/load.inc - copied unchanged from r298163, vendor/libucl/dist/tests/basic/load.inc head/contrib/libucl/tests/basic/load.res - copied unchanged from r298163, vendor/libucl/dist/tests/basic/load.res Modified: head/contrib/libucl/ChangeLog.md head/contrib/libucl/Makefile.am head/contrib/libucl/README.md head/contrib/libucl/configure.ac head/contrib/libucl/include/ucl++.h head/contrib/libucl/include/ucl.h head/contrib/libucl/lua/lua_ucl.c head/contrib/libucl/python/src/uclmodule.c head/contrib/libucl/python/test_uclmodule.py head/contrib/libucl/src/ucl_emitter.c head/contrib/libucl/src/ucl_emitter_streamline.c head/contrib/libucl/src/ucl_hash.c head/contrib/libucl/src/ucl_hash.h head/contrib/libucl/src/ucl_internal.h head/contrib/libucl/src/ucl_msgpack.c head/contrib/libucl/src/ucl_parser.c head/contrib/libucl/src/ucl_schema.c head/contrib/libucl/src/ucl_sexp.c head/contrib/libucl/src/ucl_util.c head/contrib/libucl/tests/basic.test head/contrib/libucl/tests/basic/18.in head/contrib/libucl/tests/basic/18.res head/contrib/libucl/tests/basic/2.res head/contrib/libucl/tests/basic/9.in head/contrib/libucl/tests/basic/9.res head/contrib/libucl/tests/generate.res head/contrib/libucl/tests/schema.test head/contrib/libucl/tests/schema/definitions.json head/contrib/libucl/tests/schema/ref.json head/contrib/libucl/tests/schema/refRemote.json head/contrib/libucl/tests/test_basic.c head/contrib/libucl/tests/test_generate.c head/contrib/libucl/tests/test_msgpack.c head/contrib/libucl/tests/test_schema.c head/contrib/libucl/utils/objdump.c Directory Properties: head/contrib/libucl/ (props changed) Modified: head/contrib/libucl/ChangeLog.md ============================================================================== --- head/contrib/libucl/ChangeLog.md Sun Apr 17 21:29:47 2016 (r298165) +++ head/contrib/libucl/ChangeLog.md Sun Apr 17 21:30:40 2016 (r298166) @@ -37,3 +37,31 @@ - Fixed a bug with macroes that come after an empty object - Fixed a bug in include processing when an incorrect variable has been destroyed (use-after-free) + +### Libucl 0.8.0 + +- Allow to save comments and macros when parsing UCL documents +- C++ API +- Python bindings (by Eitan Adler) +- Add msgpack support for parser and emitter +- Add Canonical S-expressions parser for libucl +- CLI interface for parsing and validation (by Maxim Ignatenko) +- Implement include with priority +- Add 'nested' functionality to .include macro (by Allan Jude) +- Allow searching an array of paths for includes (by Allan Jude) +- Add new .load macro (by Allan Jude) +- Implement .inherit macro (#100) +- Add merge strategies +- Add schema validation to lua API +- Add support for external references to schema validation +- Add coveralls integration to libucl +- Implement tests for 80% of libucl code lines +- Fix tonns of minor and major bugs +- Improve documentation +- Rework function names to the common conventions (old names are preserved for backwards compatibility) +- Add Coverity scan integration +- Add fuzz tests + +**Incompatible changes**: + +- `ucl_object_emit_full` now accepts additional argument `comments` that could be used to emit comments with UCL output \ No newline at end of file Modified: head/contrib/libucl/Makefile.am ============================================================================== --- head/contrib/libucl/Makefile.am Sun Apr 17 21:29:47 2016 (r298165) +++ head/contrib/libucl/Makefile.am Sun Apr 17 21:30:40 2016 (r298166) @@ -8,4 +8,74 @@ if LUA_SUB LUA_SUBDIR = lua endif -SUBDIRS = src tests utils doc $(LUA_SUBDIR) \ No newline at end of file +COVERAGE_INFO_FILE = $(top_builddir)/coverage.info +COVERAGE_REPORT_DIR = $(top_builddir)/coverage + +.PHONY = coverage-requirement-check clean-coverage-report + +coverage-requirement-check: + @if test ! -e $(GCOV); then \ + echo "Cannot find $(GCOV). Please install gcov."; \ + exit 1; \ + fi + +coverage: coverage-requirement-check clean-coverage coverage-build coverage-check coverage-report + @echo "Please execute 'make clean' before 'make' or 'make check' to remove instrumented object files(compiled with -O0 etc.). Note that 'make clean' also remove coverage data." + +coverage-build: coverage-requirement-check + @if test `find $(top_builddir) -name "*.gcno" | wc -l` -eq 0; then \ + echo "Start to remove old non-instrumented object files..."; \ + $(MAKE) $(AM_MAKEFLAGS) clean; \ + echo "Successfully removed old non-instrumented object files."; \ + fi + @echo "Start to build libraries with coverage options..." + $(MAKE) $(AM_MAKEFLAGS) \ + CFLAGS="$(CFLAGS) $(COVERAGE_CFLAGS) $(COVERAGE_OPTFLAGS)" \ + CXXFLAGS="$(CXXFLAGS) $(COVERAGE_CXXFLAGS) $(COVERAGE_OPTFLAGS)" \ + LDFLAGS="$(LDFLAGS) $(COVERAGE_LDFLAGS)" \ + LIBS="$(LIBS) $(COVERAGE_LIBS)" + @echo "Successfully built libraries with coverage options." + +coverage-check: coverage-requirement-check + @echo "Start to run tests with instrumented libraries..." + $(MAKE) $(AM_MAKEFLAGS) check \ + CFLAGS="$(CFLAGS) $(COVERAGE_CFLAGS) $(COVERAGE_OPTFLAGS)" \ + CXXFLAGS="$(CXXFLAGS) $(COVERAGE_CXXFLAGS) $(COVERAGE_OPTFLAGS)" \ + LDFLAGS="$(LDFLAGS) $(COVERAGE_LDFLAGS)" \ + LIBS="$(LIBS) $(COVERAGE_LIBS)" + @echo "Successfully run tests with instrumented libraries." + +coverage-lcov: coverage-check coverage-requirement-check + $(LCOV) --capture \ + --directory "$(top_builddir)/" \ + --output-file $(COVERAGE_INFO_FILE) \ + --gcov-tool $(GCOV) \ + --compat-libtool --checksum + $(LCOV) --extract $(COVERAGE_INFO_FILE) `pwd`/src/ucl_\* \ + --output-file $(COVERAGE_INFO_FILE) + +coverage-report: coverage-lcov + @echo "Start to create coverage reports..." + $(GENHTML) --prefix "$(top_srcdir)" \ + --output-directory $(COVERAGE_REPORT_DIR) \ + --title $(PACKAGE_NAME) \ + --legend --show-details \ + $(GENHTML_OPTIONS) \ + $(COVERAGE_INFO_FILE) + @echo "Successfully created coverage reports into $(COVERAGE_REPORT_DIR) directory." + +clean-coverage-report: + -rm -rf $(COVERAGE_INFO_FILE) + -rm -rf $(COVERAGE_REPORT_DIR) + +clean-coverage: clean-coverage-report + -$(LCOV) --gcov-tool $(GCOV) --zerocounters --directory $(top_builddir) + @if xargs --version 2>/dev/null; then \ + find $(top_builddir) -name "*.gcno" | xargs --no-run-if-empty rm; \ + else \ + find $(top_builddir) -name "*.gcno" | xargs rm; \ + fi + +clean-local: clean-coverage + +SUBDIRS = src tests utils doc $(LUA_SUBDIR) Modified: head/contrib/libucl/README.md ============================================================================== --- head/contrib/libucl/README.md Sun Apr 17 21:29:47 2016 (r298165) +++ head/contrib/libucl/README.md Sun Apr 17 21:30:40 2016 (r298166) @@ -1,6 +1,6 @@ # LIBUCL -[![Build Status](https://travis-ci.org/vstakhov/libucl.svg?branch=master)](https://travis-ci.org/vstakhov/libucl)[![Coverity](https://scan.coverity.com/projects/4138/badge.svg)](https://scan.coverity.com/projects/4138) +[![Build Status](https://travis-ci.org/vstakhov/libucl.svg?branch=master)](https://travis-ci.org/vstakhov/libucl)[![Coverity](https://scan.coverity.com/projects/4138/badge.svg)](https://scan.coverity.com/projects/4138)[![Coverage Status](https://coveralls.io/repos/github/vstakhov/libucl/badge.svg?branch=master)](https://coveralls.io/github/vstakhov/libucl?branch=master) **Table of Contents** *generated with [DocToc](http://doctoc.herokuapp.com/)* Modified: head/contrib/libucl/configure.ac ============================================================================== --- head/contrib/libucl/configure.ac Sun Apr 17 21:29:47 2016 (r298165) +++ head/contrib/libucl/configure.ac Sun Apr 17 21:30:40 2016 (r298166) @@ -1,7 +1,7 @@ m4_define([maj_ver], [0]) -m4_define([med_ver], [7]) -m4_define([min_ver], [3]) -m4_define([so_version], [5:0:2]) +m4_define([med_ver], [8]) +m4_define([min_ver], [0]) +m4_define([so_version], [6:0:0]) m4_define([ucl_version], [maj_ver.med_ver.min_ver]) AC_INIT([libucl],[ucl_version],[https://github.com/vstakhov/libucl],[libucl]) @@ -173,6 +173,8 @@ AC_LINK_IFELSE([ AC_MSG_WARN([Libucl references could be thread-unsafe because atomic builtins are missing]) ]) +AX_CODE_COVERAGE + AC_CONFIG_FILES(Makefile \ src/Makefile \ lua/Makefile Modified: head/contrib/libucl/include/ucl++.h ============================================================================== --- head/contrib/libucl/include/ucl++.h Sun Apr 17 21:29:47 2016 (r298165) +++ head/contrib/libucl/include/ucl++.h Sun Apr 17 21:30:40 2016 (r298166) @@ -26,7 +26,6 @@ #include #include #include -#include #include "ucl.h" @@ -120,18 +119,19 @@ public: it = std::shared_ptr(ucl_object_iterate_new (obj.obj.get()), ucl_iter_deleter()); cur.reset (new Ucl(ucl_object_iterate_safe (it.get(), true))); + if (!*cur) { + it.reset (); + cur.reset (); + } } const_iterator() {} - const_iterator(const const_iterator &other) { - it = other.it; - } + const_iterator(const const_iterator &other) = delete; + const_iterator(const_iterator &&other) = default; ~const_iterator() {} - const_iterator& operator=(const const_iterator &other) { - it = other.it; - return *this; - } + const_iterator& operator=(const const_iterator &other) = delete; + const_iterator& operator=(const_iterator &&other) = default; bool operator==(const const_iterator &other) const { @@ -264,45 +264,51 @@ public: return res; } - double number_value () const + double number_value (const double default_val = 0.0) const { - if (obj) { - return ucl_object_todouble (obj.get()); + double res; + + if (ucl_object_todouble_safe(obj.get(), &res)) { + return res; } - return 0.0; + return default_val; } - int64_t int_value () const + int64_t int_value (const int64_t default_val = 0) const { - if (obj) { - return ucl_object_toint (obj.get()); + int64_t res; + + if (ucl_object_toint_safe(obj.get(), &res)) { + return res; } - return 0; + return default_val; } - bool bool_value () const + bool bool_value (const bool default_val = false) const { - if (obj) { - return ucl_object_toboolean (obj.get()); + bool res; + + if (ucl_object_toboolean_safe(obj.get(), &res)) { + return res; } - return false; + return default_val; } - const std::string string_value () const + const std::string string_value (const std::string& default_val = "") const { - std::string res; + const char* res = nullptr; - if (obj) { - res.assign (ucl_object_tostring (obj.get())); + if (ucl_object_tostring_safe(obj.get(), &res)) { + return res; } - return res; + return default_val; } - const Ucl operator[] (size_t i) const + const Ucl at (size_t i) const { if (type () == UCL_ARRAY) { return Ucl (ucl_array_find_index (obj.get(), i)); @@ -311,15 +317,25 @@ public: return Ucl (nullptr); } - const Ucl operator[](const std::string &key) const + const Ucl lookup (const std::string &key) const { if (type () == UCL_OBJECT) { - return Ucl (ucl_object_find_keyl (obj.get(), + return Ucl (ucl_object_lookup_len (obj.get(), key.data (), key.size ())); } return Ucl (nullptr); } + + inline const Ucl operator[] (size_t i) const + { + return at(i); + } + + inline const Ucl operator[](const std::string &key) const + { + return lookup(key); + } // Serialize. void dump (std::string &out, ucl_emitter_t type = UCL_EMIT_JSON) const { @@ -328,7 +344,7 @@ public: cbdata = Ucl::default_emit_funcs(); cbdata.ud = reinterpret_cast(&out); - ucl_object_emit_full (obj.get(), type, &cbdata); + ucl_object_emit_full (obj.get(), type, &cbdata, nullptr); } std::string dump (ucl_emitter_t type = UCL_EMIT_JSON) const @@ -375,6 +391,12 @@ public: std::istreambuf_iterator()), err); } + Ucl& operator= (Ucl rhs) + { + obj.swap (rhs.obj); + return *this; + } + bool operator== (const Ucl &rhs) const { return ucl_object_compare (obj.get(), rhs.obj.get ()) == 0; @@ -388,7 +410,7 @@ public: bool operator> (const Ucl &rhs) const { return (rhs < *this); } bool operator>= (const Ucl &rhs) const { return !(*this < rhs); } - operator bool () const + explicit operator bool () const { if (!obj || type() == UCL_NULL) { return false; Modified: head/contrib/libucl/include/ucl.h ============================================================================== --- head/contrib/libucl/include/ucl.h Sun Apr 17 21:29:47 2016 (r298165) +++ head/contrib/libucl/include/ucl.h Sun Apr 17 21:30:40 2016 (r298166) @@ -107,7 +107,8 @@ typedef enum ucl_error { UCL_ENESTED, /**< Input has too many recursion levels */ UCL_EMACRO, /**< Error processing a macro */ UCL_EINTERNAL, /**< Internal unclassified error */ - UCL_ESSL /**< SSL error */ + UCL_ESSL, /**< SSL error */ + UCL_EMERGE /**< A merge error occured */ } ucl_error_t; /** @@ -147,11 +148,13 @@ typedef enum ucl_emitter { * UCL still has to perform copying implicitly. */ typedef enum ucl_parser_flags { - UCL_PARSER_DEFAULT = 0x0, /**< No special flags */ - UCL_PARSER_KEY_LOWERCASE = 0x1, /**< Convert all keys to lower case */ - UCL_PARSER_ZEROCOPY = 0x2, /**< Parse input in zero-copy mode if possible */ - UCL_PARSER_NO_TIME = 0x4, /**< Do not parse time and treat time values as strings */ - UCL_PARSER_NO_IMPLICIT_ARRAYS = 0x8 /** Create explicit arrays instead of implicit ones */ + UCL_PARSER_DEFAULT = 0, /**< No special flags */ + UCL_PARSER_KEY_LOWERCASE = (1 << 0), /**< Convert all keys to lower case */ + UCL_PARSER_ZEROCOPY = (1 << 1), /**< Parse input in zero-copy mode if possible */ + UCL_PARSER_NO_TIME = (1 << 2), /**< Do not parse time and treat time values as strings */ + UCL_PARSER_NO_IMPLICIT_ARRAYS = (1 << 3), /** Create explicit arrays instead of implicit ones */ + UCL_PARSER_SAVE_COMMENTS = (1 << 4), /** Save comments in the parser context */ + UCL_PARSER_DISABLE_MACRO = (1 << 5) /** Treat macros as comments */ } ucl_parser_flags_t; /** @@ -159,17 +162,17 @@ typedef enum ucl_parser_flags { */ typedef enum ucl_string_flags { UCL_STRING_RAW = 0x0, /**< Treat string as is */ - UCL_STRING_ESCAPE = 0x1, /**< Perform JSON escape */ - UCL_STRING_TRIM = 0x2, /**< Trim leading and trailing whitespaces */ - UCL_STRING_PARSE_BOOLEAN = 0x4, /**< Parse passed string and detect boolean */ - UCL_STRING_PARSE_INT = 0x8, /**< Parse passed string and detect integer number */ - UCL_STRING_PARSE_DOUBLE = 0x10, /**< Parse passed string and detect integer or float number */ - UCL_STRING_PARSE_TIME = 0x20, /**< Parse time strings */ + UCL_STRING_ESCAPE = (1 << 0), /**< Perform JSON escape */ + UCL_STRING_TRIM = (1 << 1), /**< Trim leading and trailing whitespaces */ + UCL_STRING_PARSE_BOOLEAN = (1 << 2), /**< Parse passed string and detect boolean */ + UCL_STRING_PARSE_INT = (1 << 3), /**< Parse passed string and detect integer number */ + UCL_STRING_PARSE_DOUBLE = (1 << 4), /**< Parse passed string and detect integer or float number */ + UCL_STRING_PARSE_TIME = (1 << 5), /**< Parse time strings */ UCL_STRING_PARSE_NUMBER = UCL_STRING_PARSE_INT|UCL_STRING_PARSE_DOUBLE|UCL_STRING_PARSE_TIME, /**< Parse passed string and detect number */ UCL_STRING_PARSE = UCL_STRING_PARSE_BOOLEAN|UCL_STRING_PARSE_NUMBER, /**< Parse passed string (and detect booleans and numbers) */ - UCL_STRING_PARSE_BYTES = 0x40 /**< Treat numbers as bytes */ + UCL_STRING_PARSE_BYTES = (1 << 6) /**< Treat numbers as bytes */ } ucl_string_flags_t; /** @@ -286,10 +289,12 @@ UCL_EXTERN ucl_object_t* ucl_object_new_ /** * Create new object with userdata dtor * @param dtor destructor function + * @param emitter emitter for userdata + * @param ptr opaque pointer * @return new object */ UCL_EXTERN ucl_object_t* ucl_object_new_userdata (ucl_userdata_dtor dtor, - ucl_userdata_emitter emitter) UCL_WARN_UNUSED_RESULT; + ucl_userdata_emitter emitter, void *ptr) UCL_WARN_UNUSED_RESULT; /** * Perform deep copy of an object copying everything @@ -306,6 +311,21 @@ UCL_EXTERN ucl_object_t * ucl_object_cop UCL_EXTERN ucl_type_t ucl_object_type (const ucl_object_t *obj); /** + * Converts ucl object type to its string representation + * @param type type of object + * @return constant string describing type + */ +UCL_EXTERN const char * ucl_object_type_to_string (ucl_type_t type); + +/** + * Converts string that represents ucl type to real ucl type enum + * @param input C string with name of type + * @param res resulting target + * @return true if `input` is a name of type stored in `res` + */ +UCL_EXTERN bool ucl_object_string_to_type (const char *input, ucl_type_t *res); + +/** * Convert any string to an ucl object making the specified transformations * @param str fixed size or NULL terminated string * @param len length (if len is zero, than str is treated as NULL terminated) @@ -642,8 +662,9 @@ UCL_EXTERN const char* ucl_object_tolstr * @param key key to search * @return object matching the specified key or NULL if key was not found */ -UCL_EXTERN const ucl_object_t* ucl_object_find_key (const ucl_object_t *obj, +UCL_EXTERN const ucl_object_t* ucl_object_lookup (const ucl_object_t *obj, const char *key); +#define ucl_object_find_key ucl_object_lookup /** * Return object identified by a key in the specified object, if the first key is @@ -655,8 +676,9 @@ UCL_EXTERN const ucl_object_t* ucl_objec * @param ... list of alternative keys to search (NULL terminated) * @return object matching the specified key or NULL if key was not found */ -UCL_EXTERN const ucl_object_t* ucl_object_find_any_key (const ucl_object_t *obj, +UCL_EXTERN const ucl_object_t* ucl_object_lookup_any (const ucl_object_t *obj, const char *key, ...); +#define ucl_object_find_any_key ucl_object_lookup_any /** * Return object identified by a fixed size key in the specified object @@ -665,8 +687,9 @@ UCL_EXTERN const ucl_object_t* ucl_objec * @param klen length of a key * @return object matching the specified key or NULL if key was not found */ -UCL_EXTERN const ucl_object_t* ucl_object_find_keyl (const ucl_object_t *obj, +UCL_EXTERN const ucl_object_t* ucl_object_lookup_len (const ucl_object_t *obj, const char *key, size_t klen); +#define ucl_object_find_keyl ucl_object_lookup_len /** * Return object identified by dot notation string @@ -674,8 +697,9 @@ UCL_EXTERN const ucl_object_t* ucl_objec * @param path dot.notation.path to the path to lookup. May use numeric .index on arrays * @return object matched the specified path or NULL if path is not found */ -UCL_EXTERN const ucl_object_t *ucl_lookup_path (const ucl_object_t *obj, +UCL_EXTERN const ucl_object_t *ucl_object_lookup_path (const ucl_object_t *obj, const char *path); +#define ucl_lookup_path ucl_object_lookup_path /** * Return object identified by object notation string using arbitrary delimiter @@ -684,8 +708,9 @@ UCL_EXTERN const ucl_object_t *ucl_looku * @param sep the sepatorator to use in place of . (incase keys have . in them) * @return object matched the specified path or NULL if path is not found */ -UCL_EXTERN const ucl_object_t *ucl_lookup_path_char (const ucl_object_t *obj, +UCL_EXTERN const ucl_object_t *ucl_object_lookup_path_char (const ucl_object_t *obj, const char *path, char sep); +#define ucl_lookup_path_char ucl_object_lookup_path_char /** * Returns a key of an object as a NULL terminated string @@ -735,6 +760,19 @@ UCL_EXTERN int ucl_object_compare (const const ucl_object_t *o2); /** + * Compare objects `o1` and `o2` useful for sorting + * @param o1 the first object + * @param o2 the second object + * @return values >0, 0 and <0 if `o1` is more than, equal and less than `o2`. + * The order of comparison: + * 1) Type of objects + * 2) Size of objects + * 3) Content of objects + */ +UCL_EXTERN int ucl_object_compare_qsort (const ucl_object_t **o1, + const ucl_object_t **o2); + +/** * Sort UCL array using `cmp` compare function * @param ar * @param cmp @@ -770,8 +808,9 @@ typedef void* ucl_object_iter_t; * while ((cur = ucl_iterate_object (obj, &it)) != NULL) ... * @return the next object or NULL */ -UCL_EXTERN const ucl_object_t* ucl_iterate_object (const ucl_object_t *obj, +UCL_EXTERN const ucl_object_t* ucl_object_iterate (const ucl_object_t *obj, ucl_object_iter_t *iter, bool expand_values); +#define ucl_iterate_object ucl_object_iterate /** * Create new safe iterator for the specified object @@ -1040,34 +1079,34 @@ UCL_EXTERN ucl_object_t* ucl_parser_get_ * @param parser parser object * @return error description */ -UCL_EXTERN const char *ucl_parser_get_error(struct ucl_parser *parser); +UCL_EXTERN const char *ucl_parser_get_error (struct ucl_parser *parser); /** * Get the code of the last error * @param parser parser object * @return error code */ -UCL_EXTERN int ucl_parser_get_error_code(struct ucl_parser *parser); +UCL_EXTERN int ucl_parser_get_error_code (struct ucl_parser *parser); /** * Get the current column number within parser * @param parser parser object * @return current column number */ -UCL_EXTERN unsigned ucl_parser_get_column(struct ucl_parser *parser); +UCL_EXTERN unsigned ucl_parser_get_column (struct ucl_parser *parser); /** * Get the current line number within parser * @param parser parser object * @return current line number */ -UCL_EXTERN unsigned ucl_parser_get_linenum(struct ucl_parser *parser); +UCL_EXTERN unsigned ucl_parser_get_linenum (struct ucl_parser *parser); /** * Clear the error in the parser * @param parser parser object */ -UCL_EXTERN void ucl_parser_clear_error(struct ucl_parser *parser); +UCL_EXTERN void ucl_parser_clear_error (struct ucl_parser *parser); /** * Free ucl parser object @@ -1076,6 +1115,42 @@ UCL_EXTERN void ucl_parser_clear_error(s UCL_EXTERN void ucl_parser_free (struct ucl_parser *parser); /** + * Get constant opaque pointer to comments structure for this parser. Increase + * refcount to prevent this object to be destroyed on parser's destruction + * @param parser parser structure + * @return ucl comments pointer or NULL + */ +UCL_EXTERN const ucl_object_t * ucl_parser_get_comments (struct ucl_parser *parser); + +/** + * Utility function to find a comment object for the specified object in the input + * @param comments comments object + * @param srch search object + * @return string comment enclosed in ucl_object_t + */ +UCL_EXTERN const ucl_object_t * ucl_comments_find (const ucl_object_t *comments, + const ucl_object_t *srch); + +/** + * Move comment from `from` object to `to` object + * @param comments comments object + * @param what source object + * @param whith destination object + * @return `true` if `from` has comment and it has been moved to `to` + */ +UCL_EXTERN bool ucl_comments_move (ucl_object_t *comments, + const ucl_object_t *from, const ucl_object_t *to); + +/** + * Adds a new comment for an object + * @param comments comments object + * @param obj object to add comment to + * @param comment string representation of a comment + */ +UCL_EXTERN void ucl_comments_add (ucl_object_t *comments, + const ucl_object_t *obj, const char *comment); + +/** * Add new public key to parser for signatures check * @param parser parser object * @param key PEM representation of a key @@ -1083,7 +1158,8 @@ UCL_EXTERN void ucl_parser_free (struct * @param err if *err is NULL it is set to parser error * @return true if a key has been successfully added */ -UCL_EXTERN bool ucl_pubkey_add (struct ucl_parser *parser, const unsigned char *key, size_t len); +UCL_EXTERN bool ucl_parser_pubkey_add (struct ucl_parser *parser, + const unsigned char *key, size_t len); /** * Set FILENAME and CURDIR variables in parser @@ -1156,8 +1232,8 @@ struct ucl_emitter_context { unsigned int indent; /** Top level object */ const ucl_object_t *top; - /** The rest of context */ - unsigned char data[1]; + /** Optional comments */ + const ucl_object_t *comments; }; /** @@ -1187,11 +1263,13 @@ UCL_EXTERN unsigned char *ucl_object_emi * @param emit_type if type is #UCL_EMIT_JSON then emit json, if type is * #UCL_EMIT_CONFIG then emit config like object * @param emitter a set of emitter functions + * @param comments optional comments for the parser * @return dump of an object (must be freed after using) or NULL in case of error */ UCL_EXTERN bool ucl_object_emit_full (const ucl_object_t *obj, enum ucl_emitter emit_type, - struct ucl_emitter_functions *emitter); + struct ucl_emitter_functions *emitter, + const ucl_object_t *comments); /** * Start streamlined UCL object emitter @@ -1280,6 +1358,9 @@ enum ucl_schema_error_code { UCL_SCHEMA_MISSING_PROPERTY,/**< one or more missing properties */ UCL_SCHEMA_CONSTRAINT, /**< constraint found */ UCL_SCHEMA_MISSING_DEPENDENCY, /**< missing dependency */ + UCL_SCHEMA_EXTERNAL_REF_MISSING, /**< cannot fetch external ref */ + UCL_SCHEMA_EXTERNAL_REF_INVALID, /**< invalid external ref */ + UCL_SCHEMA_INTERNAL_ERROR, /**< something bad happened */ UCL_SCHEMA_UNKNOWN /**< generic error */ }; @@ -1303,6 +1384,37 @@ struct ucl_schema_error { UCL_EXTERN bool ucl_object_validate (const ucl_object_t *schema, const ucl_object_t *obj, struct ucl_schema_error *err); +/** + * Validate object `obj` using schema object `schema` and root schema at `root`. + * @param schema schema object + * @param obj object to validate + * @param root root schema object + * @param err error pointer, if this parameter is not NULL and error has been + * occured, then `err` is filled with the exact error definition. + * @return true if `obj` is valid using `schema` + */ +UCL_EXTERN bool ucl_object_validate_root (const ucl_object_t *schema, + const ucl_object_t *obj, + const ucl_object_t *root, + struct ucl_schema_error *err); + +/** + * Validate object `obj` using schema object `schema` and root schema at `root` + * using some external references provided. + * @param schema schema object + * @param obj object to validate + * @param root root schema object + * @param ext_refs external references (might be modified during validation) + * @param err error pointer, if this parameter is not NULL and error has been + * occured, then `err` is filled with the exact error definition. + * @return true if `obj` is valid using `schema` + */ +UCL_EXTERN bool ucl_object_validate_root_ext (const ucl_object_t *schema, + const ucl_object_t *obj, + const ucl_object_t *root, + ucl_object_t *ext_refs, + struct ucl_schema_error *err); + /** @} */ #ifdef __cplusplus Modified: head/contrib/libucl/lua/lua_ucl.c ============================================================================== --- head/contrib/libucl/lua/lua_ucl.c Sun Apr 17 21:29:47 2016 (r298165) +++ head/contrib/libucl/lua/lua_ucl.c Sun Apr 17 21:30:40 2016 (r298166) @@ -29,6 +29,7 @@ #include "ucl_internal.h" #include "lua_ucl.h" #include +#include /*** * @module ucl @@ -149,14 +150,14 @@ ucl_object_lua_push_object (lua_State *L } /* Optimize allocation by preallocation of table */ - while (ucl_iterate_object (obj, &it, true) != NULL) { + while (ucl_object_iterate (obj, &it, true) != NULL) { nelt ++; } lua_createtable (L, 0, nelt); it = NULL; - while ((cur = ucl_iterate_object (obj, &it, true)) != NULL) { + while ((cur = ucl_object_iterate (obj, &it, true)) != NULL) { ucl_object_lua_push_element (L, ucl_object_key (cur), cur); } @@ -421,9 +422,7 @@ ucl_object_lua_fromelt (lua_State *L, in fd->idx = luaL_ref (L, LUA_REGISTRYINDEX); obj = ucl_object_new_userdata (lua_ucl_userdata_dtor, - lua_ucl_userdata_emitter); - obj->type = UCL_USERDATA; - obj->value.ud = (void *)fd; + lua_ucl_userdata_emitter, (void *)fd); } } } @@ -514,6 +513,17 @@ lua_ucl_object_get (lua_State *L, int in return *((ucl_object_t **) luaL_checkudata(L, index, OBJECT_META)); } +static void +lua_ucl_push_opaque (lua_State *L, ucl_object_t *obj) +{ + ucl_object_t **pobj; + + pobj = lua_newuserdata (L, sizeof (*pobj)); + *pobj = obj; + luaL_getmetatable (L, OBJECT_META); + lua_setmetatable (L, -2); +} + /*** * @method parser:parse_file(name) * Parse UCL object from file. @@ -629,17 +639,14 @@ static int lua_ucl_parser_get_object_wrapped (lua_State *L) { struct ucl_parser *parser; - ucl_object_t *obj, **pobj; + ucl_object_t *obj; int ret = 1; parser = lua_ucl_parser_get (L, 1); obj = ucl_parser_get_object (parser); if (obj != NULL) { - pobj = lua_newuserdata (L, sizeof (*pobj)); - *pobj = obj; - luaL_getmetatable (L, OBJECT_META); - lua_setmetatable (L, -2); + lua_ucl_push_opaque (L, obj); } else { lua_pushnil (L); @@ -806,19 +813,21 @@ lua_ucl_object_tostring (lua_State *L) } /*** - * @method object:validate(schema, path) + * @method object:validate(schema[, path[, ext_refs]]) * Validates the given ucl object using schema object represented as another * opaque ucl object. You can also specify path in the form `#/path/def` to * specify the specific schema element to perform validation. * * @param {ucl.object} schema schema object * @param {string} path optional path for validation procedure - * @return {result,err} two values: boolean result and the corresponding error + * @return {result,err} two values: boolean result and the corresponding + * error, if `ext_refs` are also specified, then they are returned as opaque + * ucl object as {result,err,ext_refs} */ static int lua_ucl_object_validate (lua_State *L) { - ucl_object_t *obj, *schema; + ucl_object_t *obj, *schema, *ext_refs = NULL; const ucl_object_t *schema_elt; bool res = false; struct ucl_schema_error err; @@ -828,15 +837,30 @@ lua_ucl_object_validate (lua_State *L) schema = lua_ucl_object_get (L, 2); if (schema && obj && ucl_object_type (schema) == UCL_OBJECT) { - if (lua_gettop (L) > 2 && lua_type (L, 3) == LUA_TSTRING) { - path = lua_tostring (L, 3); - if (path[0] == '#') { - path ++; + if (lua_gettop (L) > 2) { + if (lua_type (L, 3) == LUA_TSTRING) { + path = lua_tostring (L, 3); + if (path[0] == '#') { + path++; + } + } + else if (lua_type (L, 3) == LUA_TUSERDATA || lua_type (L, 3) == + LUA_TTABLE) { + /* External refs */ + ext_refs = lua_ucl_object_get (L, 3); + } + + if (lua_gettop (L) > 3) { + if (lua_type (L, 4) == LUA_TUSERDATA || lua_type (L, 4) == + LUA_TTABLE) { + /* External refs */ + ext_refs = lua_ucl_object_get (L, 4); + } } } if (path) { - schema_elt = ucl_lookup_path_char (schema, path, '/'); + schema_elt = ucl_object_lookup_path_char (schema, path, '/'); } else { /* Use the top object */ @@ -844,26 +868,33 @@ lua_ucl_object_validate (lua_State *L) } if (schema_elt) { - res = ucl_object_validate (schema_elt, obj, &err); + res = ucl_object_validate_root_ext (schema_elt, obj, schema, + ext_refs, &err); if (res) { lua_pushboolean (L, res); lua_pushnil (L); + + if (ext_refs) { + lua_ucl_push_opaque (L, ext_refs); + } } else { lua_pushboolean (L, res); lua_pushfstring (L, "validation error: %s", err.msg); + + if (ext_refs) { + lua_ucl_push_opaque (L, ext_refs); + } } } else { lua_pushboolean (L, res); - if (path) { - lua_pushfstring (L, "cannot find the requested path: %s", path); - } - else { - /* Should not be reached */ - lua_pushstring (L, "unknown error"); + lua_pushfstring (L, "cannot find the requested path: %s", path); + + if (ext_refs) { + lua_ucl_push_opaque (L, ext_refs); } } } @@ -872,6 +903,10 @@ lua_ucl_object_validate (lua_State *L) lua_pushstring (L, "invalid object or schema"); } + if (ext_refs) { + return 3; + } + return 2; } Modified: head/contrib/libucl/python/src/uclmodule.c ============================================================================== --- head/contrib/libucl/python/src/uclmodule.c Sun Apr 17 21:29:47 2016 (r298165) +++ head/contrib/libucl/python/src/uclmodule.c Sun Apr 17 21:30:40 2016 (r298166) @@ -2,65 +2,63 @@ #include #include -static PyObject* -_basic_ucl_type(ucl_object_t const * const obj) { - if (obj->type == UCL_INT) { - return Py_BuildValue("L", (long long)ucl_object_toint (obj)); - } - else if (obj->type == UCL_FLOAT) { - return Py_BuildValue("d", ucl_object_todouble (obj)); - } - else if (obj->type == UCL_STRING) { - return Py_BuildValue("s", ucl_object_tostring (obj)); - } - else if (obj->type == UCL_BOOLEAN) { - // maybe used 'p' here? - return Py_BuildValue("s", ucl_object_tostring_forced (obj)); - } - else if (obj->type == UCL_TIME) { - return Py_BuildValue("d", ucl_object_todouble (obj)); +static PyObject * +_basic_ucl_type (ucl_object_t const *obj) +{ + switch (obj->type) { + case UCL_INT: + return Py_BuildValue ("L", (long long)ucl_object_toint (obj)); + case UCL_FLOAT: + return Py_BuildValue ("d", ucl_object_todouble (obj)); + case UCL_STRING: + return Py_BuildValue ("s", ucl_object_tostring (obj)); + case UCL_BOOLEAN: + return ucl_object_toboolean (obj) ? Py_True : Py_False; + case UCL_TIME: + return Py_BuildValue ("d", ucl_object_todouble (obj)); } return NULL; } -static PyObject* -_iterate_valid_ucl(ucl_object_t const * obj) { +static PyObject * +_iterate_valid_ucl (ucl_object_t const *obj) +{ const ucl_object_t *tmp; ucl_object_iter_t it = NULL; tmp = obj; - while ((obj = ucl_iterate_object (tmp, &it, false))) { - - PyObject* val; + while ((obj = ucl_object_iterate (tmp, &it, false))) { + PyObject *val; val = _basic_ucl_type(obj); if (!val) { - PyObject* key = NULL; + PyObject *key = NULL; + if (obj->key != NULL) { key = Py_BuildValue("s", ucl_object_key(obj)); } - PyObject* ret; - ret = PyDict_New(); if (obj->type == UCL_OBJECT) { - val = PyDict_New(); const ucl_object_t *cur; ucl_object_iter_t it_obj = NULL; - while ((cur = ucl_iterate_object (obj, &it_obj, true))) { - PyObject* keyobj = Py_BuildValue("s",ucl_object_key(cur)); + + val = PyDict_New(); + + while ((cur = ucl_object_iterate (obj, &it_obj, true))) { + PyObject *keyobj = Py_BuildValue("s",ucl_object_key(cur)); PyDict_SetItem(val, keyobj, _iterate_valid_ucl(cur)); } - } - else if (obj->type == UCL_ARRAY) { - val = PyList_New(0); + } else if (obj->type == UCL_ARRAY) { const ucl_object_t *cur; ucl_object_iter_t it_obj = NULL; - while ((cur = ucl_iterate_object (obj, &it_obj, true))) { + + val = PyList_New(0); + + while ((cur = ucl_object_iterate (obj, &it_obj, true))) { PyList_Append(val, _iterate_valid_ucl(cur)); } - } - else if (obj->type == UCL_USERDATA) { + } else if (obj->type == UCL_USERDATA) { // XXX: this should be // PyBytes_FromStringAndSize; where is the // length from? @@ -74,13 +72,13 @@ _iterate_valid_ucl(ucl_object_t const * return NULL; } -static PyObject* -_internal_load_ucl(char* uclstr) { - PyObject* ret; - +static PyObject * +_internal_load_ucl (char *uclstr) +{ + PyObject *ret; struct ucl_parser *parser = ucl_parser_new (UCL_PARSER_NO_TIME); - bool r = ucl_parser_add_string(parser, uclstr, 0); + if (r) { if (ucl_parser_get_error (parser)) { PyErr_SetString(PyExc_ValueError, ucl_parser_get_error(parser)); @@ -88,13 +86,13 @@ _internal_load_ucl(char* uclstr) { ret = NULL; goto return_with_parser; } else { - ucl_object_t* uclobj = ucl_parser_get_object(parser); + ucl_object_t *uclobj = ucl_parser_get_object(parser); ret = _iterate_valid_ucl(uclobj); ucl_object_unref(uclobj); goto return_with_parser; } - - } else { + } + else { PyErr_SetString(PyExc_ValueError, ucl_parser_get_error (parser)); ret = NULL; goto return_with_parser; @@ -106,36 +104,151 @@ return_with_parser: } static PyObject* -ucl_load(PyObject *self, PyObject *args) { - char* uclstr; +ucl_load (PyObject *self, PyObject *args) +{ + char *uclstr; + if (PyArg_ParseTuple(args, "z", &uclstr)) { if (!uclstr) { Py_RETURN_NONE; } + return _internal_load_ucl(uclstr); } + *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***