PRIMEIRO ENVIO

2025-12-25 12:02:07 -03:00
parent ca49575224
commit c4d3bd9a86
92 changed files with 26976 additions and 1 deletions
@@ -0,0 +1,564 @@
+# Design Document: Website Cloning Platform
+
+## Overview
+
+The Website Cloning Platform is a comprehensive SaaS solution that combines advanced web scraping, intelligent crawling, and AI-powered code generation to recreate websites with near 100% visual accuracy. The platform features a microservices architecture built on Node.js with Puppeteer for browser automation, Monaco Editor for the online IDE, PostgreSQL for data management, and S3 for asset storage.
+
+## Architecture
+
+### High-Level System Architecture
+
+```mermaid
+graph TB
+    subgraph "Frontend Layer"
+        UI[React Dashboard]
+        IDE[Monaco Editor IDE]
+        Preview[Live Preview]
+    end
+    
+    subgraph "API Gateway"
+        Gateway[Express.js Gateway]
+        Auth[JWT Authentication]
+        RateLimit[Rate Limiting]
+    end
+    
+    subgraph "Core Services"
+        Crawler[Intelligent Crawler]
+        Scraper[Visual Scraper]
+        Extractor[Asset Extractor]
+        Generator[Code Generator]
+        Detector[Component Detector]
+    end
+    
+    subgraph "Processing Engine"
+        Queue[Bull Queue]
+        Workers[Worker Processes]
+        Browser[Puppeteer Pool]
+    end
+    
+    subgraph "Data Layer"
+        DB[(PostgreSQL)]
+        Cache[(Redis)]
+        Storage[(S3 Storage)]
+        CDN[CloudFront CDN]
+    end
+    
+    UI --> Gateway
+    IDE --> Gateway
+    Preview --> Gateway
+    
+    Gateway --> Crawler
+    Gateway --> Scraper
+    Gateway --> Generator
+    
+    Crawler --> Queue
+    Scraper --> Queue
+    Extractor --> Queue
+    
+    Queue --> Workers
+    Workers --> Browser
+    Workers --> DB
+    Workers --> Storage
+    
+    Storage --> CDN
+```
+
+### Microservices Architecture
+
+The platform consists of the following core microservices:
+
+1. **API Gateway Service** - Request routing, authentication, rate limiting
+2. **Crawler Service** - Website discovery and mapping
+3. **Scraper Service** - Visual element extraction
+4. **Asset Service** - File download and optimization
+5. **Generator Service** - Code generation and optimization
+6. **Component Service** - UI component detection and management
+7. **Project Service** - Project management and collaboration
+8. **IDE Service** - Online development environment
+9. **Billing Service** - Usage tracking and payment processing
+
+## Components and Interfaces
+
+### 1. Intelligent Crawler Component
+
+**Purpose:** Discovers and maps website structure using breadth-first search
+
+**Key Classes:**
+```typescript
+interface CrawlerConfig {
+  maxDepth: number;
+  maxPages: number;
+  respectRobots: boolean;
+  rateLimit: number;
+  userAgent: string;
+}
+
+interface PageInfo {
+  url: string;
+  title: string;
+  depth: number;
+  links: string[];
+  assets: AssetInfo[];
+  metadata: PageMetadata;
+}
+
+class IntelligentCrawler {
+  async crawlWebsite(startUrl: string, config: CrawlerConfig): Promise<SiteMap>;
+  async discoverLinks(page: Page): Promise<string[]>;
+  async extractPageInfo(url: string): Promise<PageInfo>;
+  async respectRateLimit(): Promise<void>;
+}
+```
+
+**Algorithms:**
+- **BFS Link Discovery:** Systematically explores website hierarchy
+- **Duplicate Detection:** Uses URL normalization and content hashing
+- **Rate Limiting:** Implements exponential backoff and concurrent request limiting
+
+### 2. Visual Scraper Component
+
+**Purpose:** Captures visual elements and layout information with high fidelity
+
+**Key Classes:**
+```typescript
+interface ScreenshotConfig {
+  viewports: Viewport[];
+  fullPage: boolean;
+  quality: number;
+  format: 'png' | 'jpeg' | 'webp';
+}
+
+interface VisualData {
+  screenshots: Screenshot[];
+  computedStyles: ComputedStyleMap;
+  layoutMetrics: LayoutMetrics;
+  animations: AnimationInfo[];
+}
+
+class VisualScraper {
+  async captureVisualData(url: string, config: ScreenshotConfig): Promise<VisualData>;
+  async executeJavaScript(page: Page): Promise<void>;
+  async captureResponsiveBreakpoints(page: Page): Promise<ResponsiveData>;
+  async extractAnimations(page: Page): Promise<AnimationInfo[]>;
+}
+```
+
+### 3. Asset Extractor Component
+
+**Purpose:** Downloads, processes, and optimizes website assets
+
+**Key Classes:**
+```typescript
+interface AssetInfo {
+  url: string;
+  type: 'image' | 'css' | 'js' | 'font' | 'media';
+  size: number;
+  hash: string;
+  dependencies: string[];
+}
+
+class AssetExtractor {
+  async extractAllAssets(pageInfo: PageInfo): Promise<AssetInfo[]>;
+  async downloadAsset(url: string): Promise<Buffer>;
+  async optimizeAsset(asset: AssetInfo, buffer: Buffer): Promise<Buffer>;
+  async resolveDependencies(cssContent: string): Promise<string[]>;
+}
+```
+
+### 4. Code Generator Component
+
+**Purpose:** Generates clean, semantic, and maintainable code
+
+**Key Classes:**
+```typescript
+interface GenerationConfig {
+  framework: 'vanilla' | 'react' | 'vue' | 'angular';
+  cssFramework: 'none' | 'tailwind' | 'bootstrap';
+  optimization: 'none' | 'basic' | 'aggressive';
+  accessibility: boolean;
+}
+
+interface GeneratedCode {
+  html: string;
+  css: string;
+  javascript: string;
+  components: ComponentDefinition[];
+  assets: AssetMapping[];
+}
+
+class CodeGenerator {
+  async generateFromVisualData(visualData: VisualData, config: GenerationConfig): Promise<GeneratedCode>;
+  async optimizeCSS(css: string): Promise<string>;
+  async generateResponsiveCSS(breakpoints: ResponsiveData): Promise<string>;
+  async createComponentStructure(elements: Element[]): Promise<ComponentDefinition[]>;
+}
+```
+
+### 5. Component Detector Component
+
+**Purpose:** Identifies and extracts reusable UI components using AI
+
+**Key Classes:**
+```typescript
+interface ComponentPattern {
+  type: 'navigation' | 'header' | 'footer' | 'card' | 'form' | 'button';
+  selector: string;
+  frequency: number;
+  variations: ComponentVariation[];
+}
+
+class ComponentDetector {
+  async detectComponents(dom: Document): Promise<ComponentPattern[]>;
+  async analyzePatterns(elements: Element[]): Promise<ComponentPattern>;
+  async extractComponentCode(pattern: ComponentPattern): Promise<ComponentDefinition>;
+  async generateComponentLibrary(patterns: ComponentPattern[]): Promise<ComponentLibrary>;
+}
+```
+
+## Data Models
+
+### Core Data Structures
+
+```typescript
+// Project Management
+interface CloneProject {
+  id: string;
+  name: string;
+  sourceUrl: string;
+  status: 'crawling' | 'processing' | 'generating' | 'completed' | 'failed';
+  createdAt: Date;
+  updatedAt: Date;
+  userId: string;
+  settings: ProjectSettings;
+  metrics: ProjectMetrics;
+}
+
+// Site Structure
+interface SiteMap {
+  rootUrl: string;
+  pages: PageInfo[];
+  assets: AssetInfo[];
+  structure: SiteStructure;
+  metadata: SiteMetadata;
+}
+
+// Component System
+interface ComponentDefinition {
+  id: string;
+  name: string;
+  type: ComponentType;
+  html: string;
+  css: string;
+  javascript?: string;
+  props: ComponentProp[];
+  instances: ComponentInstance[];
+}
+
+// User Management
+interface User {
+  id: string;
+  email: string;
+  subscription: SubscriptionTier;
+  usage: UsageMetrics;
+  projects: string[];
+  settings: UserSettings;
+}
+```
+
+### Database Schema
+
+**PostgreSQL Tables:**
+- `users` - User accounts and authentication
+- `projects` - Clone project metadata
+- `pages` - Individual page information
+- `components` - Reusable component definitions
+- `assets` - Asset metadata and references
+- `usage_logs` - Billing and analytics data
+- `collaborations` - Project sharing and permissions
+
+## Correctness Properties
+
+*A property is a characteristic or behavior that should hold true across all valid executions of a system-essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*
+
+### Crawling and Discovery Properties
+
+**Property 1: BFS Page Discovery**
+*For any* website structure, the Intelligent_Crawler should discover all reachable pages in breadth-first order, ensuring systematic exploration without missing linked content.
+**Validates: Requirements 1.1**
+
+**Property 2: Rate Limiting Compliance**
+*For any* crawling session, the system should respect configured rate limits and robots.txt directives, preventing server overload while maintaining crawling effectiveness.
+**Validates: Requirements 1.2**
+
+**Property 3: JavaScript Content Capture**
+*For any* page with dynamic content, the Multi_Layer_Processor should capture all JavaScript-generated elements in the final scraped state.
+**Validates: Requirements 1.3**
+
+**Property 4: Complete Sitemap Generation**
+*For any* crawled website, the generated sitemap should include all discovered pages with correct hierarchical relationships and no missing links.
+**Validates: Requirements 1.5**
+
+### Visual Scraping Properties
+
+**Property 5: Multi-Viewport Screenshot Capture**
+*For any* webpage and viewport configuration, the Visual_Scraper should generate screenshots for all specified viewport sizes without missing any breakpoint.
+**Validates: Requirements 2.1**
+
+**Property 6: Complete Asset Extraction**
+*For any* webpage, the Asset_Extractor should successfully download all referenced assets (images, CSS, JS, fonts, media) without missing dependencies.
+**Validates: Requirements 2.2**
+
+**Property 7: CSS Dependency Resolution**
+*For any* CSS file with @import statements, the Asset_Extractor should resolve and download all nested dependencies recursively.
+**Validates: Requirements 2.3**
+
+**Property 8: Responsive Breakpoint Capture**
+*For any* responsive webpage, the Visual_Scraper should capture all media query breakpoints and their corresponding visual states.
+**Validates: Requirements 2.4**
+
+**Property 9: Quality-Preserving Asset Optimization**
+*For any* asset optimization process, the output should maintain visual quality above acceptable thresholds while reducing file size.
+**Validates: Requirements 2.5**
+
+### Code Generation Properties
+
+**Property 10: Semantic HTML Generation**
+*For any* visual data input, the Code_Generator should produce valid, semantic HTML with proper accessibility attributes and document structure.
+**Validates: Requirements 3.1**
+
+**Property 11: CSS Modularization and Optimization**
+*For any* CSS processing task, the Code_Generator should organize styles into logical modules and remove all unused rules while preserving visual appearance.
+**Validates: Requirements 3.2**
+
+**Property 12: JavaScript Module Refactoring**
+*For any* page with inline JavaScript, the Code_Generator should extract and organize scripts into maintainable modules without breaking functionality.
+**Validates: Requirements 3.3**
+
+**Property 13: Responsive CSS Generation**
+*For any* multi-viewport visual data, the generated CSS should work correctly across all captured device sizes and breakpoints.
+**Validates: Requirements 3.4**
+
+**Property 14: Code Documentation Generation**
+*For any* complex layout structure, the Code_Generator should include explanatory comments that accurately describe the layout logic.
+**Validates: Requirements 3.5**
+
+### Component Detection Properties
+
+**Property 15: UI Pattern Recognition**
+*For any* page with repeated elements, the Component_Detector should identify all instances of the same UI pattern and mark them as reusable components.
+**Validates: Requirements 4.1**
+
+**Property 16: Component Type Classification**
+*For any* detected component, the system should correctly classify it as navigation, header, footer, card, form, or other appropriate type based on its structure and context.
+**Validates: Requirements 4.2**
+
+**Property 17: Component Parameterization**
+*For any* generated component, the code should accept parameters that allow customization of variable content while maintaining structure.
+**Validates: Requirements 4.3**
+
+**Property 18: Component Library Management**
+*For any* project, all detected components should be properly stored, organized, and accessible through the component library interface.
+**Validates: Requirements 4.4**
+
+**Property 19: Component Instance Synchronization**
+*For any* component modification, all instances of that component across the project should be updated to reflect the changes.
+**Validates: Requirements 4.5**
+
+### IDE and Collaboration Properties
+
+**Property 20: Real-Time Preview Updates**
+*For any* code modification in the IDE, the preview should update immediately to reflect the changes without manual refresh.
+**Validates: Requirements 5.2**
+
+**Property 21: Version Control Integrity**
+*For any* sequence of changes, the version control system should maintain complete history and allow rollback to any previous state without data loss.
+**Validates: Requirements 5.3**
+
+**Property 22: Collaborative Editing Conflict Resolution**
+*For any* concurrent editing session, the system should resolve conflicts automatically or provide clear resolution options without losing any user's work.
+**Validates: Requirements 5.5**
+
+### Project Management Properties
+
+**Property 23: Project Structure Organization**
+*For any* new project, the system should create a proper folder structure with organized asset management and clear file hierarchy.
+**Validates: Requirements 6.1**
+
+**Property 24: Project Sharing Access Control**
+*For any* shared project, only authorized users should have access according to their assigned permissions level.
+**Validates: Requirements 6.3**
+
+**Property 25: Deployment-Ready Export Generation**
+*For any* project export, the generated package should contain all necessary files and configurations for immediate deployment.
+**Validates: Requirements 6.4**
+
+**Property 26: Analytics Data Collection**
+*For any* project activity, the system should collect and store accurate metrics including clone accuracy and performance data.
+**Validates: Requirements 6.5**
+
+### Performance and Scalability Properties
+
+**Property 27: Parallel Processing Efficiency**
+*For any* large website processing task, the system should utilize multiple concurrent workers to reduce total processing time.
+**Validates: Requirements 7.1**
+
+**Property 28: CDN Asset Distribution**
+*For any* stored asset, the system should make it available through CDN endpoints for optimized global access.
+**Validates: Requirements 7.2**
+
+**Property 29: Intelligent Caching Behavior**
+*For any* identical content request, the system should serve from cache rather than re-processing, reducing redundant work.
+**Validates: Requirements 7.3**
+
+**Property 30: Auto-Scaling Resource Management**
+*For any* load variation, the system should automatically adjust processing resources to maintain performance while optimizing costs.
+**Validates: Requirements 7.4**
+
+**Property 31: Performance Time Constraints**
+*For any* website under 100 pages, the complete cloning process should finish within 5 minutes under normal system load.
+**Validates: Requirements 7.5**
+
+### Data Storage Properties
+
+**Property 32: PostgreSQL Data Storage**
+*For any* structured project data, the system should store it in PostgreSQL with proper schema validation and referential integrity.
+**Validates: Requirements 8.1**
+
+**Property 33: S3 Asset Organization**
+*For any* asset file, the system should store it in S3-compatible storage with logical organization and proper naming conventions.
+**Validates: Requirements 8.2**
+
+**Property 34: Backup and Recovery Functionality**
+*For any* data backup operation, the system should create complete backups that enable point-in-time recovery without data loss.
+**Validates: Requirements 8.3**
+
+**Property 35: Data Encryption Compliance**
+*For any* sensitive user data, the system should encrypt it using industry-standard methods and comply with data protection regulations.
+**Validates: Requirements 8.4**
+
+**Property 36: Data Export Portability**
+*For any* user data export request, the system should generate portable formats that allow users to migrate their data to other platforms.
+**Validates: Requirements 8.5**
+
+### Security Properties
+
+**Property 37: Multi-Factor Authentication Enforcement**
+*For any* user account, the system should require and validate multi-factor authentication before granting access to sensitive operations.
+**Validates: Requirements 9.1**
+
+**Property 38: Role-Based Permission Enforcement**
+*For any* project access attempt, the system should verify user permissions and only allow operations consistent with their assigned role.
+**Validates: Requirements 9.2**
+
+**Property 39: API Rate Limiting Protection**
+*For any* API endpoint, the system should enforce rate limits to prevent abuse while allowing legitimate usage patterns.
+**Validates: Requirements 9.3**
+
+**Property 40: Legal Compliance Warning System**
+*For any* website cloning operation, the system should display appropriate copyright and legal compliance warnings to users.
+**Validates: Requirements 9.4**
+
+**Property 41: Comprehensive Audit Logging**
+*For any* user action, the system should create detailed audit logs that enable security monitoring and compliance reporting.
+**Validates: Requirements 9.5**
+
+### Billing and Monetization Properties
+
+**Property 42: Usage-Based Billing Accuracy**
+*For any* user activity, the system should accurately track usage metrics and bill according to the appropriate subscription tier limits.
+**Validates: Requirements 10.1**
+
+**Property 43: Comprehensive Usage Monitoring**
+*For any* billable activity, the system should monitor and record pages cloned, storage used, and API calls made with precise accuracy.
+**Validates: Requirements 10.2**
+
+**Property 44: Detailed Billing Report Generation**
+*For any* billing period, the system should generate reports containing accurate usage data and billing calculations.
+**Validates: Requirements 10.3**
+
+**Property 45: Multi-Provider Payment Processing**
+*For any* payment transaction, the system should successfully process payments through multiple integrated payment providers with proper error handling.
+**Validates: Requirements 10.4**
+
+**Property 46: Free Tier Limitation Enforcement**
+*For any* free tier user, the system should enforce appropriate usage limitations while providing clear upgrade paths.
+**Validates: Requirements 10.5**
+
+## Error Handling
+
+### Error Categories and Strategies
+
+**1. Network and Connectivity Errors**
+- Implement exponential backoff for failed requests
+- Provide fallback mechanisms for CDN failures
+- Handle timeout scenarios gracefully with user feedback
+
+**2. Processing and Resource Errors**
+- Implement circuit breakers for overloaded services
+- Provide graceful degradation when resources are limited
+- Queue failed jobs for retry with appropriate delays
+
+**3. Data Validation and Integrity Errors**
+- Validate all input data at API boundaries
+- Implement database constraints and transaction rollbacks
+- Provide clear error messages for validation failures
+
+**4. Authentication and Authorization Errors**
+- Implement secure session management with proper timeouts
+- Provide clear feedback for permission denied scenarios
+- Log security-related errors for monitoring
+
+## Testing Strategy
+
+### Dual Testing Approach
+
+The platform requires both unit testing and property-based testing for comprehensive coverage:
+
+**Unit Tests:**
+- Verify specific examples and edge cases
+- Test integration points between microservices
+- Validate error conditions and boundary cases
+- Focus on concrete scenarios and known failure modes
+
+**Property-Based Tests:**
+- Verify universal properties across all inputs
+- Test system behavior with randomized data
+- Validate correctness properties from the design document
+- Ensure comprehensive input coverage through randomization
+
+### Property-Based Testing Configuration
+
+**Framework:** fast-check (JavaScript/TypeScript property-based testing library)
+**Configuration:**
+- Minimum 100 iterations per property test
+- Each test tagged with: **Feature: website-cloning-platform, Property {number}: {property_text}**
+- Custom generators for domain-specific data (URLs, HTML structures, CSS rules)
+- Shrinking enabled to find minimal failing examples
+
+**Example Test Structure:**
+```typescript
+// Feature: website-cloning-platform, Property 1: BFS Page Discovery
+fc.assert(fc.property(
+  websiteStructureGenerator(),
+  (website) => {
+    const crawler = new IntelligentCrawler();
+    const result = crawler.crawlWebsite(website.rootUrl, defaultConfig);
+    return verifyBFSOrder(result.pages) && 
+           verifyAllPagesFound(website, result.pages);
+  }
+), { numRuns: 100 });
+```
+
+### Testing Infrastructure
+
+**Test Environment:**
+- Dockerized test environment with isolated databases
+- Mock external services (S3, payment providers, CDNs)
+- Automated test data generation and cleanup
+- Performance testing with realistic load patterns
+
+**Continuous Integration:**
+- Automated test execution on all pull requests
+- Property-based test results tracked and analyzed
+- Performance regression detection
+- Security vulnerability scanning