We are KnowledgeOutOfRangeException!
A group of passionate professionals from Endava Poland. Our team consists of the following members:
 |
 |
 |
|---|---|---|
| Paweล Wawrzynek | Arkadiusz Jastrzฤbski | Krzysztof Szczฤsny |
- ๐ค Problem Description
- ๐ Feasibility Analysis
- ๐ Architecture and Design
- โ๏ธ Architecture Decision Records
Certifiable, Inc. is facing a significant increase in demand for software architecture certifications due to new regulations, as well as expansion to new markets. The company currently employs 300 expert software architects, with 5 designated experts authorized to modify certification tests. They handle around 200 candidates per week in the U.S., a number expected to grow 5-10 times with overseas expansion. The certification exam costs $800, a price fixed by the Software Architecture Licensing Board (SALB).
To manage this surge, the company is considering a redesign of their flagship system, SoftArchCert by incorporating Generative AI.
- Automate the certification process to handle increased volume.
- Maintain or improve the accuracy and reliability of the grading process.
- Reduce the workload on human graders while ensuring high standards.
This feasibility analysis evaluates the potential benefits, challenges, and overall viability of incorporating AI into the certification process.
Demand Growth:
- Expected demand for candidates without Generative AI is projected to increase significantly, reaching ~2 000 application per week by 2029.
- Based on throughput analysis, to meet this demand without generative AI, the company must either increase the contractors' headcount by 200%, or employ them full-time and still extend from 300 to 500. See ADR-001
Costs:
- The cost of employee labour, without automation, is estimated for around $1 million annually by 2029. (200% increase in headcount would triple workforce costs)
- A significant rise in spending occurs immediately after expansion in 2026.
- Recruiting and managing 500 full-time employees would require additional HR and infrastructure investment, further inflating expenses.
AI-powered automation can significantly reduce the number of full-time employees needed, limiting headcount expansion while increasing processing capacity. Instead of hiring hundreds of new employees, existing staff can focus on human oversight over AI. With AI integration, the need for a 200% increase in workforce can be eliminated, leading to substantial cost savings.
- Scalability: AI can handle a higher volume of certification requests without a proportional increase in human resources.
- Efficiency: Automated grading can significantly reduce the time required to assess tests.
- Cost Savings: Over time, the reduction in manual grading can lead to cost savings.
- Initial Investment: Implementing AI solutions requires a significant upfront investment in technology and training.
- Mitigation: Plan for phased implementation. Start with a pilot program to demonstrate value before scaling up.
- Model Accuracy: Ensuring the AI models are accurate and reliable is critical. This requires continuous monitoring and updates.
- Mitigation: Include human oversight over Generative AI to validate and correct AI outputs.
- Integration: Integrating AI into the existing system may pose technical challenges and require substantial modifications.
- Mitigation: Conduct thorough testing and have a dedicated integration team to manage the process. Use modular and scalable architecture to facilitate integration.
- User Training and Adoption: Ensuring that staff are adequately trained to use AI tools and that they adopt the new technology.
- Mitigation: Provide comprehensive training programs and ongoing support. Foster a culture of innovation and openness to new technologies.
- Technical Risks: Potential issues with AI model accuracy, integration challenges, and system performance.
- Compliance Risks: Ensuring adherence to regulatory requirements and maintaining transparency in the AI-assisted grading process.
The feasibility analysis indicates that without AI, the company will struggle to scale efficiently due to the excessive workforce and financial demands. Integrating Generative AI into the SoftArchCert system is viable and offers significant benefits in terms of scalability, efficiency, and cost savings. While there are challenges and risks, they can be mitigated through careful planning, continuous monitoring, and a balanced approach to AI and human oversight. The proposed redesign will enable Certifiable, Inc. to manage the anticipated increase in certification requests effectively and maintain high standards in the certification process.
The architecture diagram illustrates the automated grading process for Test 1. It shows high overview of how responses are processed and graded.
- Candidate complete Test 1, consisting of multiple-choice and short-answer questions.
- Responses are stored in a database as ungraded answer
- Multiple-choice questions are graded through an auto-grading system. No impact on the current process
- Short-answer responses are passed to an AI Assisted Grading Component for evaluation. Human grading process is replaced by new AI Grading
- System compiles all graded responses and stores them in a database
- Candidates are notified via email.
Test 2 is designed to assess candidates ability to create an architectural solution. Suggested approach modifies the current process by adding evaluation based on AI Assisted Grading Component
- Candidate completes Test 2 by designing an architectural solution
- Solution is uploaded to the Architectural Submissions Database
- AI analyzes the submission using predefined evaluation criteria such as design quality, correctness, etc.
- AI assigns an initial grade and feedback
- Expert reviews the submission. Hybrid approach ensures quality control while minimizing manual effort.
- Test 2 grading results are stored
- If candidate passes, their certification status is updated in the Certification Database
- Candidate receives his final result via email.
The AI-Assisted Grading is designed to automate and enhance the grading process by leveraging multiple AI models. This system reduces expert workload.
- Candidate answers are fed into the AI Prompter.
- AI Prompter formulates queries and requests grading suggestions from multiple AI Models, each with multiple different prompts. Responses are evaluated independently.
- AI Prompter takes into consideration AI Grading stats to reduce invalid responses
- Results are aggregated to generate a final grading suggestion
- Expert are asked to reviews and adjusts AI generated grades if necessary
- Grade Review/Approval update statistics for grading correctness
- Suggestions aggregator is responsible for aggregating stored grades
- AI Prompter send
Nprompt toMAI Models - Each model asynchronously send grade suggestion to AI Prompter
- AI Prompter is responsible for storing data in Grading Suggestion DB
The diagram illustrates the AI-assisted cross-check process for verifying test questions. This component provides test creators with deeper insight into questions quality. See ADR-0011 for further details.
- Component uses multiple AI models with different prompts to generate diverse answers.
- AI Models answer test questions, and these responses are submitted to AI Assisted Grading Component for evaluation
- The Question Checker retrieves question statistics from the AI Assisted Grading Component