/iXtddlmZmZddlZddlmZmZmZmZm Z m Z ddl m Z ddl mZddlmZddlmZddlmZmZmZmZdd lmZdd lmZdd lmZeZej>e Z!d e"d e#de"de#fdZ$ejKdeeejLeddedddedddeejNfde de"de"de(dee(f d Z)ejUde+e jX!ed"d#$ed%d&d'd()ed"d*$eejLeejNfd+ed,e d-e#d.e"de#de f d/Z-ej]d0e+eejLeejNfd1e"d+ede fd2Z/ejad0e+eejLeejNfd1e"de fd3Z1ejKd0eeejLeejNfd1e"de d4efd5Z2ejUd6eed"d7$ed%d&d'd8)ed"d9$eejLeejNfd1e"d-e#d.e"de#de f d:Z3ejUd;e+e jh!ed"d7$ed%d&d'd8)ed"d9$eejLeejNfd-e#d.e"de#de fd<Z5ejUd=e+ed"d7$ed%d&d'd8)ed"d>$eejLeejNfd1e"d-e#d.e"de#de f d?Z6ejUd@e+e jh!ed"d7$ed%d&d'd8)ed"d>$eejLeejNfd-e#d.e"de#de fdAZ7y)B)ListOptionalN) APIRouterDepends HTTPExceptionQuerystatusBackgroundTasks) AsyncSession)select)deps)Product) ProductCreate ProductUpdateProductResponseProductListResponse)ScrapingResponse)ProductService)ScraperService product_id headless_modetimeout_secondsenable_discoveryc vK ddlm}tjd||4d{}t j |||||d{}tjd|tjdtjd|j d dtjd |j d ddddd{y777 #1d{7swYyxYw#t$r0}tjd |d t|dYd}~yd}~wwxYww)z Background task to perform scraping after product creation. This runs asynchronously without blocking the API response. rAsyncSessionLocaluI🚀 [Background] Starting automatic first-time scraping for product ID: Nrrrru?✅ [Background] First-time scraping completed for product ID: z[Background] Scraping Summary: • Total Results: total_resultsu • Total Violations: total_violationsu3⚠️ [Background] Scraping failed for product ID : Texc_info) app.db.sessionrloggerinforscrape_product_serpget Exceptionerrorstr)rrrrrdbscraping_resultes =/var/www/html/marco-python-backend/app/api/routes/products.py_perform_background_scrapingr0s9r4 _`j_klm$& b b"$2$F$F%!1+ / %O KKYZdYef g KK8 : KK01D1D_VW1X0YZ [ KK3O4G4GHZ\]4^3_` a b b b b b b b r J:,VXY\]^Y_X`alp qqrsD9)C=C"C=C(C$BC( C=C&C=!D9"C=$C(&C=(C:.C1 /C:6C=9D9:C== D6&D1,D91D66D9/)response_model)ge d)r4le product_namez&^(product_name|msp|last_scraped_date)$)regexr,pagelimitsortBysearchcvKtj|||||d{\}}t||||S7w)zN Retrieve a list of products with pagination, sorting, and searching. )r:r;sort_byr=N)itemstotalr:r;)r get_productsr)r,r:r;r<r= current_userproductsrAs r/rBrB8sD+77 UF6OHe XUU SSs 979)r2 status_codeTz1Run browser in headless mode for initial scraping) description<z8Timeout in seconds for page load during initial scraping)r4r7rFzADiscover alternative vendors via SERP API during initial scraping product_inbackground_tasksheadlesstimeoutcXKtj||d{}tjd|jd|j d|j t|j |||tjd|j d|j dd d S7w) a Add a new product and automatically trigger scraping in the background. **Returns immediately** (HTTP 201) without waiting for scraping to complete. Scraping happens asynchronously in the background using SERP API. Parameters: - product_in: Product creation data - headless: Whether to run browser in headless mode (default: True) - timeout: Page load timeout in seconds (default: 15) - enable_discovery: Discover alternative vendors via SERP API (default: True) **Response (Immediate):** - message: Product creation success - product_id: ID of created product - scraping_status: "in_progress" - indicates scraping is running in background **Note:** Scraping continues in background. You can check results via GET /products/{id} or monitor logs for completion message: "First-time scraping completed for product ID: {id}" Nu✅ Product created:  (ID: ))rrrru3📌 Background scraping scheduled for product ID: zCProduct added successfully! Scraping is starting in the background. in_progresszuYou can start working with the product while scraping happens. Check logs or GET /products/{id} for scraping results.)messagerscraping_statusnote)rcreate_productr%r&r8idadd_taskr0)rJrKrLrMrr,rCproducts r/ add_productrYJs>#11"jAAG KK'(<(<'=VGJJ>33G  "00 ) F M!="A0A,AA0%A.&A0.A0z /scrape-allc @ Kttjtjdk(}|j |d{}|j j }|sttjdg}d} d} g} tjddtjdt|d tjdd  t|d D]"\} } d} dd l m}|}tjd d tjd| dt|d| jd| j d tjd tjd| j"tjd| j$tjd|t'j(|| j |||d{}|j+dd}|j+dd}tjd| j dtjd|tjd||j+dg}|rTtj-d| jd|D],}tj-d|d d!|d"d#d$|d%d#d.| |z } | |z } |j/||s |j9d{%tjddtjd,tjdtjd-t|tjd.t|tjd/t| tjd0| tjd1| tjdd d2d3t|d t|t|t| | | || r| nd|d4 S77I#t$ro}tj1d&| j d'|j2d()| j/| j | j|j2d*Yd}~d}~wt4$rm}tj1d&| j d't7|d)| j/| j | jt7|d*Yd}~4d}~wwxYw7$#t4$r,}tj-d+t7|Yd}~ud}~wwxYw#|rR |j9d{7w#t4$r+}tj-d+t7|Yd}~wd}~wwxYwwxYw#t4$rO}tj1d5t7|d)ttj:d6t7|d}~wwxYww)7a< Manually trigger scraping for ALL active products. Scrapes EACH product against ALL registered active vendors. For each vendor's website URL: Extract price using search and price detection. Then if discovery enabled: Google search for alternative vendors and scrape them too. This is a SYNCHRONOUS operation - waits for all products to complete before returning. Products are scraped sequentially to avoid browser and connection issues. Parameters: - headless: Whether to run browser in headless mode (default: True) - timeout: Page load timeout in seconds (default: 15) - enable_discovery: Also discover and scrape alternative vendors via Google search (default: True) Returns: - Complete results including all scraped vendors and violations for each product TN%No active products found for scrapingrdr P================================================================================u(🚀 [SCRAPE ALL] Starting scraping for z products r3r────────────────────────────────────────────────────────────────────────────────[r1] 📦 PRODUCT: rOrP Barcode:  MSP: ₹ Discovery Enabled: r scraped_countviolation_count ✅ [RESULT] Product :u • Vendors Scraped: u • Violations Found: violations 🚨 VIOLATIONS FOR  - vendor_name: ₹ scraped_pricez.3f (MSP: ₹msp ❌ [ERROR] Product r!Fr"rr8r*Warning closing session: u📊 [SCRAPE ALL] FINAL SUMMARYTotal Products: Successfully Scraped: Failed: zTotal Vendor URLs Scraped: Total Violations Found: completedScraping completed for ) r rRtotal_productssuccessful_productsfailed_productstotal_vendors_scrapedr rD failed_listru' ❌ [FATAL ERROR] Scrape All failed: Scraping failed: )r rwherer executescalarsallrrfr%r&len enumerater$rr8rVbarcoderrrgr(warningappendr*rer)r+closeHTTP_500_INTERNAL_SERVER_ERROR)rLrMrr,rCstmtrirDall_products_results total_scrapedr ridxrXfresh_dbrproduct_resultrwrxr{vher.s r/scrape_all_productsrs4 '? 4!7 8D::d# #F~~##%H 33:  MO KK$vh  KK:3x=/ST KK6(" ] %h2= MLCH; M<,. b O, auAc(m_4DWEYEYDZZ`ahakak`llmno xj* //@AB .w{{m<= 9:J9KLM(6'D'D&zz%5"*$+ ("!/ 2 2?A F "0"4"45F"J 5gjj\CD 6}oFG 77HIJ,// bA NN%>w?S?S>TTU#VW'|!M2B1C5?I[\_H``jklmrkstwjxxy'z{|.  O3 $++N;$M&nn...w= M@  d6(O$ 57 vh  &s8}o67 ,S1E-F,GHI hs?3456 1-AB ./?.@AB vhbM""05I1J0K9U!(m#&';#<"?3%2 0,.=?4 0  {$D"4!  5gjj\BII;OZ_ `&&")**$+$8$8YY(   5gjj\CF8LW[ \&&")**$+$8$8 V( /$M)B3q6('KLLMM&nn...$M)B3q6('KLLM<   @QITX Y==&s1vh/   s AXP:BX!W7C=Q4P=5C>Q3W8T2 T/ T2D)W9X=Q T, A$R3-U*3 T,?A"T'!U*'T,,U*/T22 U';!U"W"U''W*W.VV VW V< !V7 2W7V< <WW X A XXXz/{id}/scrape-serpz>Also discover alternative vendors via SERP API Google ShoppingcKtj||d{}|jsttjdt j |||||d{}|S7O7w)a Manually trigger scraping for a specific product using SERP API for discovery. Same functionality as /{id}/scrape, but uses SERP API Google Shopping Light Engine instead of Tavily API for discovering alternative vendors. Scrapes the product against all registered active vendors. When discovery is enabled, also searches SERP API Google Shopping for alternative sellers and scrapes their prices. Parameters: - id: Product ID to scrape - headless: Whether to run browser in headless mode (default: True) - timeout: Page load timeout in seconds (default: 15) - enable_discovery: if true, discover and scrape alternative vendors via SERP API (default: true) Nrcrdr)rr`r rrfrr'rhs r/trigger_scraping_serpr[su0#44R<>33G  "55 ) F M!=rkz/scrape-all-serpc Kttjtjdk(}|j |d{}|j j }|sttjdg}d} d} g} tjddtjdt|d tjdd  t|d D]\\} } d} dd l m}|}tjd d tjd| dt|d| jd| j d tjd tjd| j"tjd| j$tjdtjd|t'j(|| j |||d{}|j+dd}|j+dd}tjd| j dtjd|tjd||j+dg}|rytj-d| jd|D]Q}tj-d |j+d!d"d#|j+d$d%d&|j+d'd%dS| |z } | |z } |j/||sD |j9d{_tjddtjd.tjdtjd/t|tjd0t|tjd1t| tjd2| tjd3| tjd4tjdd d5d6t|d t|t|t| | | || r| nd|d7d8d9 S7E7#t$ro}tj1d(| j d)|j2d*+| j/| j | j|j2d,Yd}~d}~wt4$rm}tj1d(| j d)t7|d+| j/| j | jt7|d,Yd}~Kd}~wwxYw7;#t4$r,}tj-d-t7|Yd}~d}~wwxYw#|rR |j9d{7w#t4$r+}tj-d-t7|Yd}~wd}~wwxYwwxYw#t4$rO}tj1d:t7|d+ttj:d;t7|d}~wwxYww) MLCH< M<,. b O, auAc(m_4DWEYEYDZZ`ahakak`llmno xj* //@AB .w{{m<= PR 9:J9KLM(6'I'I&zz%5"*$+ ("!/ 2 2?A F (6(:(:;Mq(Q% 5gjj\CD 4]ODE =>S=TUV,// bA NN%>w?S?S>TTU#VW'Y!%% y2Q1RRWXYX]X]^motXuWvwABCBGBGHMOTBUAVVW(XYY.  $99 $++N;$M&nn...y> MB  d6(O$ :< vh  &s8}o67 ,S1E-F,GHI hs?3456 om_56 ./?.@AB MO vhbM""05I1J0KKcd!(m#&';#<"?3* 0,.=?4 0 :<  $F"4!  5gjj\BII;OZ_ `&&")**$+$8$8YY(   5gjj\CF8LW[ \&&")**$+$8$8 V( /$M)B3q6('KLLMM&nn...$M)B3q6('KLLMB   Ec!fXNY] ^==&s1vh/   s AY/R BY/!X7DR R D#R-X2VVV EX Y/R U=A$T>V; U=A"U82V;8U==V;V V8 !V3-X3V88X;X?WW WX X "!X XX XX Y,A Y''Y,,Y/)8typingrrloggingfastapirrrrr r sqlalchemy.ext.asyncior sqlalchemyr app.apir app.models.productrapp.schemas.productrrrrapp.schemas.scrapingrapp.services.product_servicerapp.services.scraper_servicerrouter getLogger__name__r%intboolr0r(get_dbget_current_userr+rBpostdictHTTP_201_CREATEDrYputr[deleter]rarj HTTP_200_OKrrrr/rs!UU/& 277    8 $rrrr rBC 34t{{+aAraC(.VW ../ TT T T  T SM T5T"S63J3JK4-`ab6pq"45xyt{{+../11%11 1  1  1L1hGD)t{{+../ 7  7 7  7* 7wt,t{{+../ 7  7 7- 7GO4t{{+../     5 ^,<=4-KLb6XY"45jkt{{+../ # ### #  #>#L]4V=O=OP4-KLb6XY"45jkt{{+../ I I I I   I QI X 64-KLb6XY"45uvt{{+../ ' ''' '  '7'T &BTBTU4-KLb6XY"45uvt{{+../ P P P P   P VP r