Ingrid Chieh Yu

• Zhou, Yan; Zhou, Baifan & Yu, Ingrid Chieh (2025). Towards Change-Instructed Action and Explanation Generation for Visual Language Action in Autonomous Driving. In Pang, Zhibo & Vyarkin, Valeriy (Ed.), 2025 IEEE 23rd International Conference on Industrial Informatics (INDIN). IEEE (Institute of Electrical and Electronics Engineers). ISSN 9798331511227. doi: 10.1109/indin64977.2025.11279112. Full text in Research Archive
• Zhou, Yan; Zhou, Baifan; Lyu, Qianhang; Waaler, Arild Torolv Søetorp & Yu, Ingrid Chieh (2025). User-Centric Question Answering with Explanation for Industrial System Information Modelling. Frontiers in Artificial Intelligence and Applications. ISSN 0922-6389. 413, p. 5392–5399. doi: 10.3233/faia251478. Full text in Research Archive Show summary

  AI holds great promise for industrial applications, but its complexity and opacity can hinder trust and limit adoption, particularly in high-stakes sectors, where transparency and accountability are essential. Question Answering with Explanation (QAE) helps address this challenge by enabling natural interaction with AI and providing justifications that allow users to assess, verify, and better understand AI-generated answers, thus improving perceived credibility. However, despite its clear benefits, QAE remains underexplored in industrial contexts. To address this gap, we employ a user-centric framework for QAE, and study it in system information modelling (SIM), a realistic and valuable industrial use case spanning disciplines, lifecycle stages, and downstream engineering tasks. Our framework integrates retrieval-augmented generation (RAG), with instruction, chain-of-thought, and few-shot prompting to guide LLMs in producing reasoning-rich explanations, and incorporates both automatic and user-perspective human evaluation. Extensive experiments on PDF and RDF datasets covering five key SIM topics demonstrate near-human-level performance. Case studies further illustrate our strengths in delivering strong argumentation, contextual evidence, and complex reasoning chains.

• Zhou, Yan; Zhou, Baifan & Yu, Ingrid Chieh (2025). NMN-BART:Generating Natural Language Explanations for Visual Question Answering. CEUR Workshop Proceedings. 3975, p. 38–48. doi: https:/ceur-ws.org/Vol-3975/paper4.pdf. Full text in Research Archive Show summary

  Visual Question Answering (VQA) is a challenging task that requires reasoning over both visual and textual information. Recently, there has been growing interest in enhancing VQA with Natural Language Explanations (NLEs) to improve transparency and trust. While existing methods leverage powerful language models for explanation generation, many score high on lexical-level text similarity rather than capturing the underlying reasoning process. In this work, we propose NMN-BART, a novel architecture that combines Neural Module Networks (NMNs) with the pretrained BART language model, using cross-modal fusion to bridge visual semantics and textual reasoning. We evaluate NMN-BART on the VQA-X dataset, where it significantly outperforms baselines on semantic-based metrics, despite lower scores on lexical similarity metrics. This suggests that our method excels in capturing the meaningful content of the explanations, rather than matching the references in wording. The case study with human evaluation further verifies our finding that our method produces semantically rich and persuasive explanations.

• Zhou, Yan; Zhou, Baifan; Li, Huajian; Lyu, Qianhang; Qu, Yuanwei & Waaler, Arild Torolv Søetorp [Show all 7 contributors for this article] (2025). Dataset for Industrial Question Answering with Explanation and Scalable Ensemble Generation. In Long, Guodong; Blumestein, Michale & Chang, Yi (Ed.), WWW '25: Companion Proceedings of the ACM on Web Conference 2025. Association for Computing Machinery (ACM). ISSN 9798400713316. p. 825–828. doi: 10.1145/3701716.3715310. Full text in Research Archive Show summary

  The digital and green transition under Industry 4.0 has accelerated the adoption of AI in industries such as manufacturing, energy, and mining. Question Answering with Explanation (QAE), as a way of human interaction with AI, is crucial for enhancing transparency and trust in high-stakes industrial applications. However, industrial QAE remains underexplored due to the lack of publicly available, high-quality datasets, hindered by the need for expert effort and corporate restrictions. To this end, we introduce PANDAX ( https://doi.org/10.5281/zenodo.14510798 ), the first open-source industrial QAE dataset, and SEG, a scalable method for generating high-quality QAE datasets using LLMs. PANDAX focuses on three key topics of industrial system information: partonomy, functionality, and parameters, across critical domains such as green technology and cooling systems. SEG ensures scalability and quality through ensemble generation, majority voting, expert ranking, etc. The human evaluation validates PANDAX's high quality, positioning it as a valuable resource for advancing QAE techniques, benchmarking language technologies, and supporting research in explainable AI for industrial systems.

• Din, Crystal Chang; Dridi, Charaf Eddine; Motzfeldt, Ida; Pun, Violet Ka I; Stolz, Volker & Yu, Ingrid Chieh (2025). Modular soundness checking of feature model evolution plans. Theoretical Computer Science. ISSN 0304-3975. 1054. doi: 10.1016/j.tcs.2025.115451. Full text in Research Archive Show summary

  Feature model evolution plans (FMEPs) describe how feature models for software product lines (SPLs) evolve over time. While different feature models can exist for different points in time over the lifetime of the product line, an FMEP describes how to compute a feature model for a given time point. SPLs capitalise on the variability and reusability of the software through combining optional and mandatory features. As business requirements change over time, FMEPs should support intermediate update. A plan hence contains updates to an initial model by adding, deleting, moving or changing elements at different points in time, in line with the evolving business requirements on the SPL, potentially affecting feature models that should be derived in the future from the plan. A recurring challenge in maintaining FMEPs is that updates may lead to inconsistent intermediate feature models, most notably so-called paradoxes. A paradox may not materialise at the first point in time an update on the plan is performed to obtain a particular feature model, but may only in combination with a later modification prescribed by the plan create a structurally invalid model. Correspondingly, a single modification to a plan may require multiple checks over the liftetime of the affected elements to rule out paradoxes. Current approaches require the analysis from the point in time an update is applied to an FMEP throughout the entire lifetime of the plan. In this paper, we define a so-called interval-based feature model (IBFM) to represent FMEPs, with a precise definition of spatial and temporal scopes that narrow the time interval and the sub-models that an update can affect. We propose a rule system for updating IBFMs, and also prove the soundness of the proposed rules and show their modularity, i.e., that each rule operates strictly within its temporal and spatial scopes. We have conducted a detailed evaluation on our modular approach and present the experimental results, which show that we outperform an existing linear approach.

• Segal, Meirav; George, Anne-Marie; Yu, Ingrid Chieh & Dimitrakakis, Christos (2024). Better Luck Next Time: About Robust Recourse in Binary Allocation Problems. Communications in Computer and Information Science (CCIS). ISSN 1865-0929. 2155, p. 374–394. doi: 10.1007/978-3-031-63800-8_19. Full text in Research Archive
• Motzfeldt, Ida Sandberg; Yu, Ingrid Chieh; Din, Crystal Chang; Pun, Violet Ka I & Stolz, Volker (2023). Modular Soundness Checking of Feature Model Evolution Plans. Lecture Notes in Computer Science (LNCS). ISSN 0302-9743. 14446, p. 417–437. doi: 10.1007/978-3-031-47963-2_25. Full text in Research Archive Show summary

  A software product line (SPL) is a family of closely related software systems which capitalizes on the variability and reusability of the software products and can be formalised by a feature model. Feature model evolution plans (FMEP) capture the current SPL as well as the planned evolution of the SPL to ensure successful long-term development. As business requirements often change, FMEPs should support interme- diate update. This modification may cause paradoxes in an FMEP, e.g., a node left without a parent, making the plan impossible to realise. Current tools exist to validate FMEPs, but require analysing the entire plan even when a modification affects only small parts of it. Hence, there is a need for a method that detects such paradoxes in a more efficient way. In this paper, we present an interval based feature model (IBFM), a representation for FMEPs, that allows local reasoning to validates only the parts of the plan affected by changes. We define operations for updating the FMEPs and the preconditions under which they preserve soundness, i.e., absence of paradoxes, and show the correctness of the method.

• Segal, Meirav; George, Anne-Marie; Yu, Ingrid Chieh & Dimitrakakis, Christos (2023). Robust Recourse for Binary Allocation Problems. Neurips 2023 workshop on XAI in Action: Past, Present, and Future Applications. Full text in Research Archive
• Sæther, Eirik Halvard; Yu, Ingrid Chieh & Din, Crystal Chang (2023). Semantics-Based Version Control for Feature Model Evolution Plans. NIKT: Norsk IKT-konferanse for forskning og utdanning. ISSN 1892-0713. 1. Full text in Research Archive Show summary

  A software product line (SPL) models closely related software systems by capitalizing on the high similarity of the products by organizing them into common and variable parts. To ensure successful long-term development, it is beneficial to not just capture the current software product line, but the planned evolution of the SPL as well. Evolution planning of an SPL is often a dynamic, changing process due to changes in product requirements. In addition, planning is typically a collaborative effort with multiple engineers working separately and independently of each other. To improve development, their individual contributions would need to be unified. This can be a complex task, especially without proper synchronization tools. In this paper, we provide a semantics-based merge algorithm for evolution plans. Given two versions of an evolution plan and the common evolution plan they are derived from, the merge algorithm attempts to merge all the different changes from both versions. The merge algorithm will be an essential component in a version control system, allowing several contributors to unify their versions into a sound evolution plan.

• Rasouli, Peyman & Yu, Ingrid Chieh (2022). CARE: coherent actionable recourse based on sound counterfactual explanations. International Journal of Data Science and Analytics (JDSA). ISSN 2364-415X. doi: 10.1007/s41060-022-00365-6. Full text in Research Archive
• Rasouli, Peyman; Yu, Ingrid Chieh & Jimenez-Ruiz, Ernesto (2022). Interpreting Categorical Data Classifiers using Explanation-based Locality. IEEE International Conference on Data Mining Workshops, ICDMW. ISSN 2375-9232. p. 163–170. doi: 10.1109/ICDMW58026.2022.00030. Full text in Research Archive
• Yu, Ingrid Chieh; Pene, Irina; Din, Crystal Chang; Karlsen, Leif Harald; Nguyen, Chi Mai & Stahl, Oliver [Show all 7 contributors for this article] (2022). Subsurface Evaluation Through Multi-scenario Reasoning. In Patel, Daniel (Eds.), Interactive Data Processing and 3D Visualization of the Solid Earth. Springer Nature. ISSN 9783030907150. p. 325–355. doi: 10.1007/978-3-030-90716-7_10. Full text in Research Archive
• Lebesbye, Torgeir; Mauro, Jacopo; Turin, Gianluca & Yu, Ingrid Chieh (2021). Boreas – A Service Scheduler for Optimal Kubernetes Deployment. Lecture Notes in Computer Science (LNCS). ISSN 0302-9743. p. 221–237. doi: 10.1007/978-3-030-91431-8_14. Full text in Research Archive Show summary

  The advent of cloud computing radically changed the way organisations operate their applications and allows them to achieve high availability of services at affordable cost. Most cloud-computing platforms fostered Kubernetes for their container orchestration and service management. The scheduler is a key component of Kubernetes, as it is responsible for finding the placement of new service containers when they are deployed. The default scheduler is very fast, although often suboptimal. This can lead to inefficient placement of services, or more severely, inability to deploy. We present a custom Kubernetes scheduler, dubbed Boreas, which is designed to evaluate bursts of deployment requests concurrently. Boreas finds the optimal placements for service containers with their deployment constraints by utilising a configuration optimiser. Results show that Boreas is able to find placements where the default Kubernetes scheduler fails, wasting less computing resources, or proving that no feasible deployment solution is possible.

• Rasouli, Peyman & Yu, Ingrid Chieh (2021). Analyzing and Improving the Robustness of Tabular Classifiers using Counterfactual Explanations, 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA). IEEE (Institute of Electrical and Electronics Engineers). ISSN 9781665443371. p. 1286–1293. doi: 10.1109/ICMLA52953.2021.00209. Full text in Research Archive Show summary

  Recent studies have revealed that Machine Learning (ML) models are vulnerable to adversarial perturbations. Such perturbations can be intentionally or accidentally added to the original inputs, evading the classifier’s behavior to misclassify the crafted samples. A widely-used solution is to retrain the model using data points generated by various attack strategies. However, this creates a classifier robust to some particular evasions and can not defend unknown or universal perturbations. Counterfactual explanations are a specific class of post-hoc explanation methods that provide minimal modification to the input features in order to obtain a particular outcome from the model. In addition to the resemblance of counterfactual explanations to the universal perturbations, the possibility of generating instances from specific classes makes such approaches suitable for analyzing and improving the model’s robustness. Rather than explaining the model’s decisions in the deployment phase, we utilize the distance information obtained from counterfactuals and propose novel metrics to analyze the robustness of tabular classifiers. Further, we introduce a decision boundary modification approach using customized counterfactual data points to improve the robustness of the models without compromising their accuracy. Our framework addresses the robustness of black-box classifiers in the tabular setting, which is considered an under-explored research area. Through several experiments and evaluations, we demonstrate the efficacy of our approach in analyzing and improving the robustness of black-box tabular classifiers.

• Rasouli, Peyman & Yu, Ingrid Chieh (2021). Explainable Debugger for Black-box Machine Learning Models. Proceedings of the International Joint Conference on Neural Networks. ISSN 2161-4393. doi: 10.1109/IJCNN52387.2021.9533944. Full text in Research Archive Show summary

  The research around developing methods for debugging and refining Machine Learning (ML) models is still in its infancy. We believe employing tailored tools in the development process can help developers in creating more trustworthy and reliable models. This is particularly essential for creating black-box models such as deep neural networks and random forests, as their opaque decision-making and complex structure prevent detailed investigations. Although many explanation techniques provide interpretability in terms of predictive features for a mispredicted instance, it would be beneficial for a developer to find a partition of the training data that significantly influences the anomaly. Such responsible partitions can be subjected to data visualization and data engineering in the development phase to improve the model's accuracy. In this paper, we propose a systematic debugging framework for the development of ML models that guides the data engineering process using the model's decision boundary. Our approach finds the influential neighborhood of anomalous data points using observation-level feature importance and explains them via a novel quasi-global explanation technique. It is also equipped with a robust global explanation approach to reveal general trends and expose potential biases in the neighborhoods. We demonstrate the efficacy of the devised framework through several experiments on standard data sets and black-box models and propose various guidelines on how the framework's components can be practically useful from a developer's perspective.

• Rasouli, Peyman & Yu, Ingrid Chieh (2020). EXPLAN: Explaining Black-box Classifiers using Adaptive Neighborhood Generation. Proceedings of the International Joint Conference on Neural Networks. ISSN 2161-4393. doi: 10.1109/IJCNN48605.2020.9206710. Full text in Research Archive Show summary

  Defining a representative locality is an urgent challenge in perturbation-based explanation methods, which influences the fidelity and soundness of explanations. We address this issue by proposing a robust and intuitive approach for EXPLaining black-box classifiers using Adaptive Neighborhood generation (EXPLAN). EXPLAN is a module-based algorithm consisted of dense data generation, representative data selection, data balancing, and rule-based interpretable model. It takes into account the adjacency information derived from the black-box decision function and the structure of the data for creating a representative neighborhood for the instance being explained. As a local model-agnostic explanation method, EXPLAN generates explanations in the form of logical rules that are highly interpretable and well-suited for qualitative analysis of the model’s behavior. We discuss fidelity-interpretability trade-offs and demonstrate the performance of the proposed algorithm by a comprehensive comparison with state-of-the-art explanation methods LIME, LORE, and Anchor. The conducted experiments on real-world data sets show our method achieves solid empirical results in terms of fidelity, precision, and stability of explanations.

• Lin, Jia-Chun; Lee, Ming-Chang; Yu, Ingrid Chieh & Johnsen, Einar Broch (2020). A Configurable and Executable Model of Spark Streaming on Apache YARN. International Journal of Grid and Utility Computing (IJGUC). ISSN 1741-847X. 11(2), p. 185–195. doi: 10.1504/IJGUC.2020.105531. Full text in Research Archive Show summary

  Abstract: Streams of data are produced today at an unprecedented scale. Efficient and stable processing of these streams requires a careful interplay between the parameters of the streaming application and of the underlying stream processing framework. Today, finding these parameters happens by trial and error on the complex, deployed framework. This paper shows that high-level models can help to determine these parameters by predicting and comparing the performance of streaming applications running on stream processing frameworks with different configurations. To demonstrate this approach, this paper considers Spark Streaming, a widely used framework to leverage data streams on the fly and provide real-time stream processing. Technically, we develop a configurable and executable model to simulate both the streaming applications and the underlying Spark stream processing framework. Furthermore, we model the deployment of Spark Streaming on Apache YARN, which is a popular open-source distributed software framework for big data processing. We show that the developed model provides a satisfactory accuracy for predicting performance by means of empirical validation.

• Hoff, Adrian; Nieke, Michael; Seidl, Christoph; Sæther, Eirik; Motzfeldt, Ida & Din, Crystal Chang [Show all 8 contributors for this article] (2020). Consistency-Preserving Evolution Planning on Feature Models. In Lopez-Herrejon, Roberto Erick (Eds.), SPLC'20: 24th ACM International Systems and Software Product Line Conference, Montreal, Quebec, Canada, October 19-23, 2020, Volume A. Association for Computing Machinery (ACM). ISSN 9781450375696. p. 8:1–8:12. doi: 10.1145/3382025.3414964. Full text in Research Archive Show summary

  A software product line (SPL) enables large-scale reuse in a family of related software systems through configurable features. SPLs represent a long-term investment so that their ongoing evolution becomes paramount and requires careful planning. While existing approaches enable to create an evolution plan for an SPL on feature-model (FM) level, they assume the plan to be rigid and do not support retroactive changes. In this paper, we present a method that enables to create and retroactively adapt an FM evolution plan while preventing undesired impacts on its structural and logical consistency. This method is founded in structural operational semantics and linear temporal logic. We implement our method using rewriting logic, integrate it within an FM tool suite and perform an evaluation using a collection of existing FM evolution scenarios.

• Gkolfi, Anastasia; Johnsen, Einar Broch; Kristensen, Lars Michael & Yu, Ingrid Chieh (2020). Model checking starvation for resource-aware active objects with coloured petri nets. CEUR Workshop Proceedings. 2651, p. 68–85. Full text in Research Archive Show summary

  Dynamic resource provisioning is an important driver for pay-on-demand cloud computing. Virtualized resources open for resource awareness, such that applications may use resource management strategies to modify their deployment resource consumption at run-time. The ABS language supports the modeling of deployment decisions and resource management for active objects. An important property in this context is to ensure that the resource management does not lead to starvation of the executing objects. In previous work, we have formally translated the semantics of the ABS language into a parameterized Coloured Petri Net (CPN) model, such that any ABS program can be represented by setting the initial marking accordingly. In this paper, we characterize starvation using Computation Tree Logic (CTL), and demonstrate how CTL model checking of the CPN encoding of the ABS model, in combination with path finding, can be used to detect starvation and synthesize load balancers that guarantee starvation freedom.

• Gkolfi, Anastasia; Din, Crystal Chang; Johnsen, Einar Broch; Kristensen, Lars Michael; Steffen, Martin & Yu, Ingrid Chieh (2019). Translating active objects into colored Petri nets for communication analysis. Science of Computer Programming. ISSN 0167-6423. 181, p. 1–26. doi: 10.1016/j.scico.2019.04.002. Full text in Research Archive
• Rasouli, Peyman & Yu, Ingrid Chieh (2019). Meaningful Data Sampling for a Faithful Local Explanation Method. Lecture Notes in Computer Science (LNCS). ISSN 0302-9743. p. 28–38. doi: 10.1007/978-3-030-33607-3_4. Full text in Research Archive Show summary

  Data sampling has an important role in the majority of local explanation methods. Generating neighborhood samples using either the Gaussian distribution or the distribution of training data is a widely-used procedure in the tabular data case. Generally, this approach has several weaknesses: first, it produces a uniform data which may not represent the actual distribution of samples; second, disregarding the interaction between features tends to create unlikely samples; and third, it may fail to define a compact and diverse locality for the sample being explained. In this paper, we propose a sampling methodology based on observation-level feature importance to derive more meaningful perturbed samples. To evaluate the efficiency of the proposed approach we applied it to the LIME explanation method. The conducted experiments demonstrate considerable improvements in terms of fidelity and explainability.

• Din, Crystal Chang; Karlsen, Leif Harald; Pene, Irina; Stahl, Oliver; Yu, Ingrid Chieh & Østerlie, Thomas (2019). Geological Multi-scenario Reasoning. NIKT: Norsk IKT-konferanse for forskning og utdanning. ISSN 1892-0713. Full text in Research Archive Show summary

  In the oil and gas industry, during exploration prospect assessment, explorationists rely on ad hoc manual work practices and tools for developing and communicating multiple hypothetical geological scenarios of the prospect. This leaves them with little efficient means to make the fullest use of state of the art digital technologies to communicate and systematically compare and assess different hypothetical geological scenarios before deciding which scenario to pursue. In this paper, we present a formal framework for geological multi-scenario reasoning, a novel tool-based method for geologically oriented subsurface evaluation. The methodology applies formal methods and logic-based techniques to subsurface evaluation and expresses interpretive uncertainty as discrete scenarios with branches of potential alternatives. This framework consists of (i) a proto-scenario generator that takes user observations and geological evidence as input and generates semantically valid initial states based on formalized geological knowledge in first-order logic (ii) geological processes formalized as a rewrite theory that are executable in Maude. By applying geological rewrite rules onto the proto-scenarios, we are able to assist explorationists with multi-scenario generation and reasoning beyond human capacity.

• Nieke, Michael; Mauro, Jacopo; Seidl, Christoph; Thum, Thomas; Yu, Ingrid Chieh & Franzke, Felix (2018). Anomaly Analyses for Feature-Model Evolution. In Wyk, Eric Van & Rompf, Tiark (Ed.), Proceedings of the 17th ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences. ACM Publications. ISSN 9781450360456. p. 188–201. doi: 10.1145/3278122.3278123. Full text in Research Archive
• Mauro, Jacopo; Tarifa, Silvia Lizeth Tapia & Yu, Ingrid Chieh (2018). Automatic Parameter Optimisation of Service Quality and Resource Usage. NIKT: Norsk IKT-konferanse for forskning og utdanning. ISSN 1892-0713. Full text in Research Archive Show summary

  https://ojs.bibsys.no/index.php/NIK/article/view/504/430

• Din, Crystal Chang; Johnsen, Einar Broch; Owe, Olaf & Yu, Ingrid Chieh (2018). A modular reasoning system using uninterpreted predicates for code reuse. Journal of Logical and Algebraic Methods in Programming. ISSN 2352-2208. 95, p. 82–102. doi: 10.1016/j.jlamp.2017.11.004. Full text in Research Archive Show summary

  This paper proposes a modular proof system based on uninterpreted predicates. The proposed proof system allows modular reasoning about programs with an open-world assumption, which goes beyond behavioral subtyping. The proof system enables modular reasoning about languages with very flexible code reuse mechanisms, such as traits and deltas in the context of object-oriented programming. Whereas related work on incremental proof systems prove soundness in terms of internal consistency, this paper establishes both soundness and relative completeness of the proposed proof system by relating it to a standard proof system for a simple object-oriented language. The applicability of the approach is demonstrated on different code reuse mechanisms: unrestricted class inheritance, delta-oriented programming, and trait-based programming.

• Mauro, Jacopo; Nieke, Michael; Seidl, Christoph & Yu, Ingrid Chieh (2018). Context-aware reconfiguration in evolving software product lines. Science of Computer Programming. ISSN 0167-6423. 163, p. 139–159. doi: 10.1016/j.scico.2018.05.002. Full text in Research Archive
• Røst, Thomas Brox; Seidl, Christoph; Yu, Ingrid Chieh; Damiani, Ferruccio; Johnsen, Einar Broch & Chesta, Cristina (2018). HyVar: Scalable Hybrid Variability for Distributed Evolving Software Systems. Communications in Computer and Information Science (CCIS). ISSN 1865-0929. 824, p. 159–163. doi: 10.1007/978-3-319-79090-9_12. Full text in Research Archive Show summary

  The HyVar project (www.hyvar-project.eu/) proposes a development framework for continuous and individualized evolution of distributed software applications running on remote devices in heterogeneous environments, focusing on the automotive domain. The framework combines variability modeling and software reuse from software product lines with formal methods and software upgrades and can be integrated in existing software development processes. HyVar’s objectives are: (O1) To develop a Domain Specific Variability Language (DSVL) and tool chain to support software variability for highly distributed applications; (O2) to develop a cloud infrastructure that exploits software variability as described in the DSVL to track the software configurations deployed on remote devices and to enable (i) the collection of data from the devices to monitor their behavior; and (ii) secure and efficient customized updates; (O3) to develop a technology for over-the-air updates of distributed applications, which enables continuous software evolution after deployment on complex remote devices that incorporate a system of systems; and (O4) to test HyVar’s approach as described in the above objectives in an industry-led demonstrator to assess in quantifiable ways its benefits. The end of the project is approaching and we are close to reaching all the objectives. In this paper, we present the integrated tool chain, which combines formal reuse through software product lines with commonly used industrial practices, and supports the development and deployment of individualized software adaptations. We also describe the main benefits for the stakeholders involved.

• Schlatte, Rudolf; Johnsen, Einar Broch; Mauro, Jacopo; Tarifa, Silvia Lizeth Tapia & Yu, Ingrid Chieh (2018). Release the Beasts: When Formal Methods Meet Real World Data. Lecture Notes in Computer Science (LNCS). ISSN 0302-9743. 10865, p. 107–121. doi: 10.1007/978-3-319-90089-6_8. Full text in Research Archive
• Lin, Jia-Chun; Lee, Ming-Chang; Yu, Ingrid Chieh & Johnsen, Einar Broch (2018). Modeling and simulation of spark streaming. Advanced Information Networking and Applications. ISSN 1550-445X. 2018-May, p. 407–413. doi: 10.1109/AINA.2018.00068. Full text in Research Archive Show summary

  As more and more devices connect to Internet of Things, unbounded streams of data will be generated, which have to be processed "on the fly" in order to trigger automated actions and deliver real-time services. Spark Streaming is a popular realtime stream processing framework. To make efficient use of Spark Streaming and achieve stable stream processing, it requires a careful interplay between different parameter configurations. Mistakes may lead to significant resource overprovisioning and bad performance. To alleviate such issues, this paper develops an executable and configurable model named SSP (stands for Spark Streaming Processing) to model and simulate Spark Streaming. SSP is written in ABS, which is a formal, executable, and object-oriented language for modeling distributed systems by means of concurrent object groups. SSP allows users to rapidly evaluate and compare different parameter configurations without deploying their applications on a cluster/cloud. The simulation results show that SSP is able to mimic Spark Streaming in different scenarios.

• Gkolfi, Anastasia; Johnsen, Einar Broch; Kristensen, Lars Michael & Yu, Ingrid Chieh (2018). Using coloured Petri nets for resource analysis of active objects. Lecture Notes in Computer Science (LNCS). ISSN 0302-9743. 11222 LNCS, p. 156–174. doi: 10.1007/978-3-030-02146-7_8. Full text in Research Archive Show summary

  Pay-on-demand resource provisioning is an important driver for cloud computing. Virtualized resources in cloud computing open for resource awareness, such that applications may contain resource management strategies to modify their deployment and reduce resource consumption. The ABS language supports the modelling of deployment decisions and resource management for active objects. In this paper, the semantics of ABS is captured directly as a Coloured Petri Net (CPN) model capable of representing any ABS program by an appropriate initial marking. We define an abstraction relation between the CPN model and the language semantics such that markings of the CPN model become abstract ABS configurations. We use a CPN model checker as an abstract interpreter to investigate resource distribution and starvation problems for deployed active objects in ABS.

• Gkolfi, Anastasia; Din, Crystal Chang; Johnsen, Einar Broch; Steffen, Martin & Yu, Ingrid Chieh (2017). Translating Active Objects into Colored Petri Nets for Communication Analysis. Lecture Notes in Computer Science (LNCS). ISSN 0302-9743. p. 84–99. doi: 10.1007/978-3-319-68972-2_6. Full text in Research Archive
• Lin, Jia-Chun; Mauro, Jacopo; Røst, Thomas Brox & Yu, Ingrid Chieh (2017). A model-Based Scalability Optimization Methodology for Cloud Applications. Proceedings of the IEEE. ISSN 0018-9219. p. 163–170. doi: 10.1109/SC2.2017.32. Full text in Research Archive
• Gkolfi, Anastasia; Johnsen, Einar Broch; Kristensen, Lars Michael & Yu, Ingrid Chieh (2017). Resource Management of Cloud-Aware Programs using Coloured Petri Nets. In Walden, Marina (Eds.), Proceedings of the 29th Nordic Workshop on Programming Theory. Turku Centre for Computer Science. ISSN 9789521236082. p. 31–33. Full text in Research Archive
• Hestvik, Magnus Røed; Mauro, Jacopo & Yu, Ingrid Chieh (2017). CaSPL-gen: a Context-aware Software Product Line benchmark generator. NIKT: Norsk IKT-konferanse for forskning og utdanning. ISSN 1892-0713. Full text in Research Archive
• Mauro, Jacopo; Seidl, Christoph; Nieke, Michael & Yu, Ingrid Chieh (2017). Anomaly Detection and Explanation in Context-Aware Software Product Lines. In Benavides, David & Ruiz-Cortés, Antonio (Ed.), SPLC '17 Proceedings of the 21st International Systems and Software Product Line Conference - Volume B. Association for Computing Machinery (ACM). ISSN 9781450351195. p. 18–21. doi: 10.1145/3109729.3109752. Full text in Research Archive
• Nieke, Michael; Mauro, Jacopo; Seidl, Christoph & Yu, Ingrid Chieh (2016). User profiles for context-aware reconfiguration in software product lines. Lecture Notes in Computer Science (LNCS). ISSN 0302-9743. 9953 LNCS, p. 563–578. doi: 10.1007/978-3-319-47169-3_44. Full text in Research Archive
• Damiani, Ferruccio; Seidl, Christoph & Yu, Ingrid Chieh (2016). Introduction to the track on variability modeling for scalable software evolution. Lecture Notes in Computer Science (LNCS). ISSN 0302-9743. 9953 LNCS, p. 423–432. doi: 10.1007/978-3-319-47169-3_35. Full text in Research Archive
• Johnsen, Einar Broch; Lin, Jia-Chun & Yu, Ingrid Chieh (2016). Comparing AWS Deployments Using Model-Based Predictions. Lecture Notes in Computer Science (LNCS). ISSN 0302-9743. 9953, p. 482–496. doi: 10.1007/978-3-319-47169-3_39. Full text in Research Archive
• Bubel, Richard; Damiani, Ferruccio; Hähnle, Reiner; Johnsen, Einar Broch; Owe, Olaf & Schaefer, Ina [Show all 7 contributors for this article] (2016). Proof Repositories for Compositional Verification of Evolving Software Systems - Managing Change When Proving Software Correct. Lecture Notes in Computer Science (LNCS). ISSN 0302-9743. 9960, p. 130–156. doi: 10.1007/978-3-319-46508-1_8. Full text in Research Archive
• Lin, Jia-Chun; Yu, Ingrid Chieh; Johnsen, Einar Broch & Lee, Ming-Chang (2016). ABS-YARN: A Formal Framework for Modeling Hadoop YARN Clusters. In Stevens, Perdita & Wąsowski, Andrzej (Ed.), Fundamental Approaches to Software Engineering: 19th International Conference, FASE 2016, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2016, Eindhoven, The Netherlands, April 2-8, 2016, Proceedings. Springer. ISSN 9783662496657. p. 49–65. doi: 10.1007/978-3-662-49665-7_4. Full text in Research Archive
• Li, Jingyue; Qeriqi, Altin; Steffen, Martin & Yu, Ingrid Chieh (2016). Automatic translation from FBD-PLC-programs to NuSMV for model checking safety-critical control systems. NIKT: Norsk IKT-konferanse for forskning og utdanning. ISSN 1892-0713. Full text in Research Archive
• Johnsen, Einar Broch; Pun, Ka I; Steffen, Martin; Tarifa, Silvia Lizeth Tapia & Yu, Ingrid Chieh (2016). Meeting Deadlines, Elastically. In Petre, Luigia & Sekerinski, Emil (Ed.), From Action Systems to Distributed Systems: The Refinement Approach. CRC Press. ISSN 9781498701587. p. 99–111. doi: 10.1201/b20053-11. Full text in Research Archive
• Rabbi, Fazle; Lamo, Yngve; Yu, Ingrid Chieh & Kristensen, Lars Michael (2016). WebDPF: A Web-based Metamodelling and Model Transformation Environment. In Hammoudi, Slimane; Pires, Luís Ferreira; Selic, Bran & Desfray, Philippe (Ed.), Proceedings of the 4th International Conference on Model-Driven Engineering and Software Development. SciTePress. ISSN 9789897581687. p. 87–98. doi: 10.5220/0005686900870098. Full text in Research Archive
• Rabbi, Fazle; Lamo, Yngve & Yu, Ingrid Chieh (2016). Towards a categorical approach for meta-modelling epistemic game theory. In Baudry, Benoit & Combemale, Benoit (Ed.), MODELS '16: Proceedings of the ACM/IEEE 19th International Conference on Model Driven Engineering Languages and Systems. Association for Computing Machinery (ACM). ISSN 9781450343213. p. 57–64. doi: 10.1145/2976767.2976771. Full text in Research Archive
• Mauro, Jacopo; Nieke, Michael; Seidl, Christoph & Yu, Ingrid Chieh (2016). Context Aware Reconfiguration in Software Product Lines. In Schaefer, Ina; Alves, Vander & Almeida, Eduardo Santana de (Ed.), Proceedings of the Tenth International Workshop on Variability Modelling of Software-intensive Systems. Association for Computing Machinery (ACM). ISSN 9781450340199. p. 41–48. doi: 10.1145/2866614.2866620. Full text in Research Archive
• Rabbi, Fazle; Lamo, Yngve; Yu, Ingrid Chieh & Kristensen, Lars Michael (2016). Diagrammatic development of domain specific modelling languages with WebDPF. International Journal of Information System Modeling and Design. ISSN 1947-8186. 7(3), p. 93–114. doi: 10.4018/IJISMD.2016070105. Full text in Research Archive
• Yu, Ingrid Chieh & Berg, Henning (2015). A Formalisation of Analysis-based Model Migration. In Hammoudi, Slimane; Pires, Luis Ferreira; Desfray, Philippe & Filipe, Joaquim (Ed.), MODELSWARD 2015 - Proceedings of the 3rd International Conference on Model-Driven Engineering and Software Development, ESEO, Angers, Loire Valley, France, 9-11 February, 2015. SciTePress. ISSN 9789897580833. p. 86–98. doi: 10.5220/0005240900860098. Full text in Research Archive
• Dovland, Johan; Johnsen, Einar Broch; Owe, Olaf & Yu, Ingrid Chieh (2015). A proof system for adaptable class hierarchies. Journal of Logical and Algebraic Methods in Programming. ISSN 2352-2208. 84(1), p. 37–53. doi: 10.1016/j.jlamp.2014.09.001. Full text in Research Archive
• Rabbi, Fazle; Lamo, Yngve; Yu, Ingrid Chieh & Kristensen, Lars Michael (2015). A diagrammatic approach to model completion. CEUR Workshop Proceedings. 1500, p. 56–65. Full text in Research Archive Show summary

  Metamodelling plays an important role in model-driven engineering as it can be used to define domain-specific modelling languages. During the modelling phase, software designers encode domain knowledge into models which may include both structural and behavioral aspects of a system. In this paper we propose a diagrammatic approach to aid the software designer to complete partial models and thereby reduce modelling effort. We introduce a declarative approach where we define completion rules that are executed by the use of model transformations. We also study the termination of such model transformation systems and provide sufficient conditions for termination by generalizing existing work on termination of model transformation systems.

• Rabbi, Fazle; Lamo, Yngve; Yu, Ingrid Chieh & Kristensen, Lars Michael (2015). Towards a Multi Metamodelling Approach for Developing Distributed Healthcare Applications. NIKT: Norsk IKT-konferanse for forskning og utdanning. ISSN 1892-0713. Full text in Research Archive Show summary

  Model Driven Engineering (MDE) uses formal methods to build mathematically rigorous models of complex systems. Metamodelling plays an important role in MDE as it is used to specify domain specific modelling languages. However, the potential of metamodelling has not been fully explored. Current approaches of MDE are often at a low level of abstraction and lack domain concepts for specifying behavior. In previous work, we proposed a multi metamodelling approach that captures the complexity of systems by using a metamodelling hierarchy, built from individually defined metamodels, each capturing different aspects of a healthcare domain. In this paper, we focus on modelling distributed healthcare applications and present an example from the healthcare domain. We address certain modelling aspects related to distributed applications such as process modelling, using message passing communication, and coordination of processes and resources.

• Owe, Olaf & Yu, Ingrid Chieh (2014). Deadlock detection of active objects with synchronous and asynchronous method calls. NIKT: Norsk IKT-konferanse for forskning og utdanning. ISSN 1892-0713. 3. doi: 10.1016/j.entcs.2004.06.012. Full text in Research Archive
• Damiani, Ferruccio; Dovland, Johan; Johnsen, Einar Broch; Owe, Olaf; Schaefer, Ina & Yu, Ingrid Chieh (2012). A Transformational Proof System for Delta-Oriented Programming, SPLC'12 Proceedings of the 16th International Software Product Line Conference - Volume 2. ACM Publications. ISSN 9781450310956. p. 53–60. doi: 10.1145/2364412.2364422. Full text in Research Archive
• Dovland, Johan; Johnsen, Einar Broch & Yu, Ingrid Chieh (2012). Tracking Behavioral Constraints during Object-Oriented Software Evolution. Lecture Notes in Computer Science (LNCS). ISSN 0302-9743. 7609, p. 253–268. doi: 10.1007/978-3-642-34026-0_19. Full text in Research Archive
• Dovland, Johan; Johnsen, Einar Broch; Owe, Olaf & Yu, Ingrid Chieh (2011). A Proof System for Adaptable Class Hierarchies. In Petterson, Paul & Cristina, Seceleanu (Ed.), Proceedings of the 23rd Nordic Workshop on Programming Theory, NWPT'11. Mälardalen University. ISSN 9789521224782. p. 33–34. doi: 10.1016/j.jlamp.2014.09.001. Full text in Research Archive
• Johnsen, Einar Broch; Kyas, Marcel & Vibe, Ingrid Chieh Yu De (2009). Dynamic Classes: Modular Asynchronous Evolution of Distributed Concurrent Objects. Lecture Notes in Computer Science (LNCS). ISSN 0302-9743. 5850, p. 596–611. Full text in Research Archive
• Johnsen, Einar Broch & Vibe, Ingrid Chieh Yu De (2008). Backwards type analysis of asynchronous method calls. Journal of Logic and Algebraic Programming. ISSN 1567-8326. 77(1-2), p. 40–59. doi: 10.1016/j.jlap.2008.05.004. Full text in Research Archive
• Hagalisletto, Moen Anders; Bjørk, Joakim; Vibe, Ingrid Chieh Yu De & Enger, P (2007). Constructing and refining large-scale railway models represented by Petri nets. IEEE transactions on systems, man and cybernetics. Part C, Applications and reviews. ISSN 1094-6977. 37, p. 444–460. doi: 10.1109/TSMC.2007.897323. Full text in Research Archive Show summary

  A new method for rapid construction of large-scale executable railway models is presented. Computer systems for railway systems suffer from poor integration and lack of explicit understanding of the large amount of static and dynamic information in the railway. In this paper we give solutions to both problems. It is shown how a component-oriented approach makes it easy to construct and refine basic railway models by effective methods, such that a variety of models with important properties can be maintained within the same framework. Basic railway nets are refined into several new kinds; nets that are safe, permit collision detection, include time, and are sensitive to its surroundings. Since the underlying implementation language is Petri Nets, large expressibility is combined with simplicity, and in addition the analysis of the behaviour of railway models come gently.

• Johnsen, Einar Broch & Yu, Ingrid Chieh (2006). Backwards Type Analysis of Asynchronous Method Calls, Proceedings for the 18th Nordic Workshop on Programming Theory (NWPT'06). Hàskolinn i Reykjavik. Full text in Research Archive
• Johnsen, Einar Broch; Owe, Olaf & Yu, Ingrid Chieh (2006). Creol: A type-safe object-oriented model for distributed concurrent systems. Theoretical Computer Science. ISSN 0304-3975. 365, p. 23–66. doi: 10.1016/j.tcs.2006.07.031. Full text in Research Archive
• Yu, Ingrid Chieh; Johnsen, Einar Broch & Owe, Olaf (2006). Type-safe runtime class upgrades in Creol. Lecture Notes in Computer Science (LNCS). ISSN 0302-9743. 4037, p. 202–217. Full text in Research Archive

View all works in NVA

• Yu, Ingrid Chieh & Berg, Henning (2015). A Framework for Metamodel Composition and Adaptation with Conformance-Preserving Model Migration. Springer. ISBN 9783319278681. 23 p. Full text in Research Archive
• Hagalisletto, Anders Moen & Yu, Ingrid Chieh (2004). Large scale construction of railroad models from specifications. IEEE (Institute of Electrical and Electronics Engineers). ISBN 9780780385672. 8 p. Full text in Research Archive Show summary

  In this paper we present techniques for constructing large scale models of railroads from a simple graphical specification language,using Petri Nets. A prototype tool based on this analysis has been implemented, allowing the user to write specifications and generate large Petri Nets automatically. A sufficient amount of algebra for composition is developed in order to exploit the advantages of both modern GUI programming and Petri Nets.

View all works in NVA

• Ramezanifarkhani, Toktam; Owe, Olaf & Yu, Ingrid Chieh (2021). Leadership in Academia. Full text in Research Archive
• Yu, Ingrid Chieh; Cameron, David B.; Vølstad, Ann & Larsen, Åshild Hanne (2020). Industrial mentoring for junior researchers: An enabler for personal development and innovation. Full text in Research Archive Show summary

  The SIRIUS Centre for Research-Based Innovation, based at the University of Oslo, is a collaborative organization that addresses challenges of digitalisation in, and beyond, the oil and gas industry. It aims to produce innovation that solves operational challenges in industry through application of high-quality com-puter science research. The centre brings together academics from the three universities, two research in-stitutes, large companies in the oil and gas sector, IT vendors, both global and local, and a group of spe-cialized companies. It is a complex task to generate industrial innovation and excellent research simultaneously. The centre brings together a diverse group of participants, with widely differing backgrounds and motivations. Effort must be made to bridge the gaps between our academic researchers and their industrial counterparts. As part of this effort, SIRIUS ran a mentoring program for junior academics in 2017 and 2018. This paper describes the program, its results and practical experience that was derived from the program. Each of the authors brings their own perspective to the paper. The lead author is the deputy director of the centre. She defined the objectives, led and ran the mentoring program. The second author was the manager in the centre with responsibility for relationships with the centre’s industrial partners. He was the sponsor of the program. The third author works for AFF, a consultancy that prepared the content of the program and facilitated the meetings in the program. The final author is one of the mentors. She is also a C-level executive in the energy company Equinor.

• Johnsen, Einar Broch; Lin, Jia-Chun & Yu, Ingrid Chieh (2016). AWS deployments using model-based predictions. Full text in Research Archive
• Lin, Jia-Chun; Yu, Ingrid Chieh; Johnsen, Einar Broch & Lee, Ming-Chang (2016). ABS-YARN: A Formal Framework for Modeling Hadoop YARN Clusters. Full text in Research Archive
• Owe, Olaf; Lin, Jia-Chun & Yu, Ingrid Chieh (2015). A Formal Framework Supporting Unrestricted Software Changes in Object-Oriented Concurrent Systems. Full text in Research Archive
• Rabbi, Fazle; Lamo, Yngve; Kristensen, Lars Michael & Yu, Ingrid Chieh (2015). A generalization of termination conditions for partial model completion. Full text in Research Archive
• Yu, Ingrid Chieh (2015). Gjer smarte bilar smartare. [Journal]. Uniforum. Full text in Research Archive
• Rabbi, Fazle; Lamo, Yngve; Yu, Ingrid Chieh & Kristensen, Lars Michael (2014). Towards a Diagrammatic Rewriting System for Model Completion. Full text in Research Archive
• Motzfeldt, Ida Sandberg; Yu, Ingrid Chieh & Din, Crystal Chang (2021). Modular Soundness Checking of Feature Model Evolution Plans. Universitetet i Oslo. Full text in Research Archive Show summary

  A software product line (SPL) is a family of closely related software systems which capitalizes on the reusability and variability of the software products. An SPL can be modelled using a feature model, a tree-like structure from which all the configurations of the SPL can be derived. Large projects such as an SPL require long-term planning, and plans for SPLs may also be defined in terms of feature models, called feature model evolution plans (FMEP). An FMEP gives information about what a feature model looks like at each stage of the plan. As business requirements often change, FMEPs should support intermediate change. Such changes may cause paradoxes in an FMEP, e.g. a node left without a parent, making the plan impossible to realise. The complex nature of FMEPs makes detecting paradoxes by hand impractical. Current tools exist to validate FMEPs, but require analysis of the entire plan even when a modification affects only small parts of it. For larger FMEPs, this is inefficient. Thus, there is a need for a method which detects such paradoxes in a more efficient way. In this thesis, we present a representation for FMEPs, called an interval-based feature model (IBFM). This representation enables local validation, by which we mean validating only the parts of the plan that are affected by the change. We define operations for updating an IBFM, and methods for detecting paradoxes resulting from an operation. Moreover, we give a proof of correctness for the method and an implementation as proof of concept. Using these methods, it is possible to create an efficient verification tool for modification of FMEPs. This may be used as basis for a productive SPL planning tool.

• Sæther, Eirik Halvard; Yu, Ingrid Chieh & Din, Crystal Chang (2021). Three-Way Semantic Merge for Feature Model Evolution Plans. Universitetet i Oslo. Full text in Research Archive Show summary

  A software product line (SPL) models closely related software systems by capitalizing on the high similarity of the products by organizing them in common and variable parts. Software engineers explicitly encode similarities and differences of an SPL by defining it in terms of a feature model. In order to ensure successful long-term development, it is beneficial to not just capture the current software product line, but the planned evolution of the SPL as well. Evolution planning of an SPL is often a dynamic, changing process, due to changes in product requirements. In addition, planning is typically a collaborative effort with multiple engineers working separately and independently of each other. To improve development, their individual contributions would need to be synchronized and unified. This can be a complex task, especially without proper synchronization tools. In this thesis, we develop a merge tool for evolution plans. The essence of the tool is a three-way merge algorithm. Given two different versions of an evolution plan, together with the common evolution plan they are derived from, the merge algorithm attempts to merge all the different changes from both versions. If the evolution plans are unifiable, the algorithm succeeds and yields the merged result containing the changes from both versions. However, if the changes are conflicting by breaking the structure or semantics of evolution plans, the algorithm terminates and reports the reason for failure. The three-way merge algorithm will act as an essential component in a version control system, allowing several contributors to synchronize their individual versions into a unified evolution plan.

• Lebesbye, Torgeir; Yu, Ingrid Chieh & Mauro, Jacopo (2019). Boreas – Reducing Resource Usage Through Optimized Kubernetes Scheduling. Universitetet i Oslo. Full text in Research Archive
• Rabbi, Fazle; Yu, Ingrid Chieh; Lamo, Yngve & Kristensen, Lars Michael (2017). MDE TECHNIQUES FOR MODELING AND ANALYSIS OF COMPLEX SOFTWARE SYSTEMS. Reprosentralen, University of Oslo. Full text in Research Archive
• Berg, Henning & Yu, Ingrid Chieh (2017). Generic Metamodel Refactoring with Automatic Detection of Applicability and Co-evolution of Artefacts. Universitetet i Oslo. ISSN 9788273684264. Full text in Research Archive Show summary

  Refactoring is an essential operation in software engineering, with the purpose of improving the structural qualities of software. An emerging trend in software engineering is the use of model-driven approaches in which models are used as first-class entities. The problem domain concepts used in models are formalised by a metamodel. Refactoring of metamodels is an important task in increasing the extensibility and maintainability of metamodels. However, changing the definition of a metamodel comes at the cost of introducing inconsistencies in modelling artefacts that are defined relatively to the metamodel. In this report, we present a framework extension for refactoring of metamodels that allows re-establishing consistency between the artefacts and the metamodel. We also discuss how refactoring patterns can be applied automatically on metamodels. The framework has been formally defined by a deductive system.

• Johnsen, Einar Broch; Pun, Ka I; Steffen, Martin; Tarifa, Silvia Lizeth Tapia & Yu, Ingrid Chieh (2015). Meeting Deadlines, Elastically. Universitetet i Oslo. ISSN 9788273684080. Full text in Research Archive
• Yu, Ingrid Chieh (2010). Type Safety for Distributed Concurrent Objects and Runtime Upgrades. Universitetet i Oslo. Full text in Research Archive
• Johnsen, Einar Broch; Owe, Olaf & Yu, Ingrid Chieh (2005). Creol: A Type-Safe Object-Oriented Model for Distributed Concurrent Systems. Institutt for informatikk, Universitetet i Oslo. Full text in Research Archive

View all works in NVA